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PREFACE 


SPONSORS .  The  Workshop  on  Flight  Testing  to  Identify  Pilot 
Workload  and  Pilot  Dynamics  was  held  at  the  Edwards  AFB 
Officer's  Club  19-21  January  1982.  The  workshop  was  spon¬ 
sored  by  the  Air  Force  Flight  Test  Center  in  conjunction 
with  the  Flight  Dynamics  Laboratory,  NASA-Dryden,  the 
American  Institute  of  Aeronautics  and  Astronautics,  and  the 
Human  Factors  Society. 

NEED  .  Advances  in  avionics  and  control  systems  have  in¬ 
creased  the  mission  capability  of  military  and  civil, 
fixed-  and  rotary-winged  aircraft  .  The  challenge  to  de¬ 
signers  of  advanced  aircraft  systems  now  is  to  make  the 
crewmember's  job.  easier,  i.e.,  to  allow  him  to  precisely 
control  the  aircraft  and  succesfully  perform  the  mission 
while  maintaining  adequate  mental  and  physical  reserve 
capacity  to  handle  emergencies .  This  design  problem  calls 
for  all  parties  involved  in  the  research,  development, 
test,  and  evaluation  of  advanced  aircraft  to  understand 
the  elements  that  influence  the  efficient  interaction  be¬ 
tween  the  pilot  and  aircraft .  Fundamental  to  this  under¬ 
standing  are  quantitative  measures  of  pilot  workload  and 
performance,  and  descriptive  models  of  the  pilot's  dynamics. 
Currently  such  measures  and  models  are  not  mature  enough 
for  reliable  use  in  the  testing  and  development  of  advanced 
aircraft  .  A  forum  is  needed  to  gather  the  proper  expertise 
in  the  areas  of  flight  testing,  workload  measurement, 
pilot-in-the-loop  dynamics,  and  applications  so  that  the 
state-of-the-art  in  these  areas  can  be  understood  and 
approaches  can  be  recommended  that  will  develop  flight-worthy 
measures  of  pilot  workload  and  flight-validated  models  of 
pilot  dynamics  .  The  present  workshop,  the  fourth  in  an 
annual  series  exploring  specialized  areas  of  flight  test¬ 
ing,  was  structured  to  meet  this  need. 

OBJECTIVE .  The  objective  of  this  workshop  was  to  bring 
together  technical  experts  working  with  various  measures  of 
pilot  workload  and  performance  and  with  various  models  of 
pilot  dynamics  to  meet  face-to-face  with  the  flight  test 
and  applications  community  and  together  to  define: 

1  .  The  need  for  and  applications  of  flight-worthy 
measures  of  pi’lot  workload  and  flight-validated  models  of 
pilot  dynamics, 

2  .  The  state-of-the-art  and  current  problems  to  be 
overcome  in  order  to  measure  pilot  workload  or  task  per¬ 
formance  in  flight  or  to  tailor  flight  tests  to  identify 
pilot,  dynamics ,  and 

3  .  Recommended  ways  to  solve  these  technical  problems  . 
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PROGRAM .  The  program  consisted  of  three  days  of  presen¬ 
tations  and  informal  discussions  tailored  to  the  objectives 
of  the  workshop.  The  first  day  was  intended  as  an  overview 
of  the  many  facets  of  pilot  workload  and  pilot  dynamics . 
The  second  day  featured  presentations  dealing  with  flight 
testing  for  pilot  workload.  The  final  day  agenda  addressed 
topics  related  to  flight  testing  to  identify  pilot  dynamics 
and  task  performance . 

PROCEEDINGS .  This  volume  is  divided  into  three  sections 
corresponding  to  the  three  days  of  the  workshop .  It  is  a 
compilation  of  written  material  submitted  by  individual 
authors  for  publication.  A  few  papers  not  presented  at 
the  workshop  are  included  because  of  their  direct  bearing 
on  the  workshop  theme.  The  editors  wish  to  thank  all  the 
authors  who  contributed  to  the  quality  and  success  of  the 
workshop . 

ACKNOWLEDGEMENT .  The  editors  are  very  grateful  for  the 
assistance  given  by  the  individuals  named  below  both  in  the 
conduct  of  the  workshop  and  in  the  compilation  of  these 
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The  findings  of  the  USAF  Night  Attack  Workload  Steering 
Group  will  be  of  interest  to  the  readers  of  this  report. 
They  will  be  published  this  year  in  three  volumes  by  the 
Aeronautical  Systems  Division  (ASD/ENEC) ,  Wright-Patterson 
AFB,  Ohio  45433  as  ASD-TR-5002. 
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SECTION  I 


OVERVIEW  OF  PILOT  WORKLOAD,  PILOT  DYNAMICS, 
AND  FLIGHT  TEST  REQUIREMENTS 
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A  PILOT’S  PERSPECTIVE  ON  WORKLOAD 
IN  SINGLE  SEAT  FIGHTERS 

DAVID  W.  MILAM,  Lt  Col,  USAF 
Director,  F-16  Combined  Test  Force 
Air  Force  Flight  Test  Center 


INTRODUCTION 

The  recent  loss  of  four  fine  young  pilots  at  a  Thunderbird  practice 
in  Nevada  makes  our  job  of  analyzing  pilot  workload  even  more  critical. 

It  is  obvious  to  me  that  we  have  a  lot  to  learn  about  the  manner  in  which 
the  brain  functions  when  the  pilot  is  doing  high  gain  tasks.  We  should 
set  a  goal  for  ourselves  of  learning  enough  about  mental  and  physical 
workload  to  make  the  probability  very  remote  of  ever  having  a  similar 
accident . 

Pilot  workload  in  single  seat  fighters  is  obviously  increasing.  That 
pilot  workload  includes  physical  workload  and  mental  workload.  The  physical 
workload  has  steadily  increased.  In  the  past,  the  F-4  could  pull  7.33  g’s. 
Today  the  F-16  can  sustain  9  g’s,  and  in  the  future  the  AFTI/F-16  will  add 
2  lateral  g's  to  the  aircraft  maneuvering  capability.  The  Impact  of  that 
physical  workload  on  mental  workload  and  task  accomplishment  needs  to  be 
determined.  Mental  workload  demands  have  increased  by  a  gigantic  amount  due 
to  the  increase  in  the  amount  of  information  displayed  to  the  pilot  and  the 
increased  complexity  of  modern  systems.  The  F-4  has  a  radar  &  lead  computing 
sight.  The  F-16  has  a  heads  up  display,  stores  control  panel,  fire  control 
computer,  and  sophisticated  radar.  We  are  also  forced  to  fly  faster  and 
lower  to  evade  sophisticated  ground  based  threats.  The  new  MSIP  III  F-16s 
will  have  two  multi-function  displays,  wide  field  of  view  hud,  internal  ECM, 
global  positioning  system,  navigation  pod,  and  an  increased  weapons  capacity 
of  radar  missiles  and  30  mm  gun.  The  next  generation  fighter  will  have  to  be 
designed  to  kill  tanks  at  night  in  European  weather  and  battle  for  air 
superiority  against  overwhelming  odds.  Pilot  workload  must  be  one  of  the 
major  factors  to  consider  as  new  weapons  systems  are  developed. 

The  decision  process  for  designing  and  building  new  systems  is  complex 
and  good  decisions  are  based  on  a  good  analysis  of  the  tradeoffs  between 
cost  and  benefit.  In  order  to  determine  the  relative  balance  of  gain  and 
cost,  there  are  several  impacts  which  need  to  be  known: 

Cost 

Performance 

Maintainability 

Reliability 

Pilot  Workload 

If  a  system  cost  nothing,  improves  performance,  decreases  workload, 
and  does  not  change  maintainability  or  reliability  then  the  decision  to 
incorporate  is  easy.  That  combination  never  occurs,  however,  and  we  need 
to  accept  the  responsibility  to  provide  a  complete  picture  of  cost  and 
benefit  tradeoffs  to  decision  maker. 
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Cost  can  be  determined  with  at  least  some  degree  of  confidence  by 
signing  a  contract.  Hidden  costs  always  exist  but  within  reasonable 
limits,  costs  can  be  determined. 

Performance  can  be  determined  also.  During  flight  test  or  simulation 
we  can  assess  the  total  response  of  the  pilot’  and  the  vehicle  by  measuring 
bomb  scores  or  evaluating  landing  accuracy. 

Maintainability  is  not  easily  measured  and  we  do  not  always  structure 
tests  to  evaluate  maintainability.  There  are,  however,  some  traditional 
methods  of  measuring  the  total  impact  by  measuring  the  man-hours  required 
to  keep  a  system  flying. 

Reliability  is  a  top  priority.  The  long  term  impact  on  lift  cycle  cost 
and  maintenance  man-hours  is  significant.  Additionally,  the  decrease  in 
compat  capability  of  aircraft  having  sophisticated  systems  is  large.  The 
combat  effectiveness  of  an  F-15  without  an  operable  radar  is  very  low.  It 
is  possible,  however,  to  measure  mean  time  between  failures  (MTBF)  and  have 
a  general  picture  of  reliability. 

Cost,  performance,  maintainability,  and  reliability  have  some  definite 
standards  of  measurement  which  allow  comparision  for  good  decision  making, 
however,  pilot  workload  does  not  presently  have  that  definitive  measurement. 

It  is  clear  that  future  decisions  need  to  be  based  on  an  accurate,  complete 
workload  measurement.  The  methods  of  measurement  which  seem  to  be  possible 
are  quantitative  measurements  and  pilot  qualitative  opinion. 

CHALLENGES  TO  WORKLOAD  DETERMINATION 

There  are  several  considerations  which  I  believe  impact  pilot  performance 
and  workload.  I  have  included  in  the  Appendix  a  discussion  of  flying  as  a 
skill  task  where  training,  hand-eye  coordinating,  conscious  versus  subconscious 
information  processing,  and  individual  techniques  are  Involved.  These 
concepts  form  the  foundation  upon  which  I  will  base  the  following  challenges, 
as  I  see  them,  to  workload  determination. 

Research  to  measure  mental  workload  as  well  as  physical  workload  is 
very  important  because  pilot  qualitative  opinion  is  not  always  consistent. 
Remember,  pilots  are  unique.  Just  as  we  measure  the  short  period  damping  of 
an  airplane,  it  would  be  nice  to  measure  physical  and  mental  workload  with  a 
laboratory  proven,  measureable  parameter.  It  would  be  nice  to  have  some 
unobtrusive  physiological  measurement  to  quantify  pilot  workload.  Two 
different  measurements  (such  as  heart  rate  and  eye  movement)  will  probably 
be  needed  to  distinguish  between  physical  and  mental  workload.  This 
distinction  is  needed  by  decision-makers  in  the  design  of  new  systems. 

For  example,  the  separate  effects  of  side-acceleration  restraints  and 
various  display  formats  will  need  to  be  evaluated  during  AFTI/F-16  bombing 
tests . 


In  reality,  however,  the  measurement  of  workload  is  still  accomplished 
primarily  by  pilot  qualitative  opinion.  A  pilot  estimates  his  workload. 

For  that  workload  estimation  to  be  meaningful,  it  is  mandatory  to  take  the 
average  of  several  pilots.  For  the  estimation  of  workload  to  be  consistent, 
then,  we  need  to  strictly  control  the  tests  and  use  the  same  rating  scale. 
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Mr  Tom  Twisdale  has  written  many  documents  on  how  to  accomplish  flying 
qualities  evaluations.  He  structures  tests  in  a  precise  manner,  measures 
performance,  and  collects  pilot  ratings.  His  primary  (but  not  his  only) 
rating  scale  is  the  Cooper -Harper  scale  which  combines  performance  and 
workload.  After  our  recent  flight  tests  to  evaluate  avionics  subsystems 
there  is  a  question  in  my  mind  about  the  utility  of  measuring  performance 
and  workload  together  on  the  Cooper -Harper  rating  scale.  Most  test  pilots 
do  not  use  the  C-H  scale  properly  but  tend  to  give  a  relative  numerical 
rating  from  1  to  10  where  perfect  is  1  and  uncontrollable  is  10.  A  modified 
subjective  rating  scale  method  should  be  developed  to  allow  the  pilot  to 
evaluate  workload  and  performance  separately.  Such  a  scale  could  be 
validated  during  flying  qualities  evaluations  along  side  the  Cooper -Harper 
scale.  The  scale  would  then  be  of  great  value  to  the  evaluation  of  avionics, 
flight  control,  and  other  subsystem  changes  and  their  effects  on  total 
system  performance. 

Workload  measurements,  whether  subjective  or  objective,  should  be 
available  much  earlier  in  the  systems  design  process  so  that  design  options 
can  be  intelligently  considered.  If  cost,  performance,  maintainability, 
reliability,  and  workload  impacts  are  known  very  early  on  all  design  options, 
then  good  decisions  may  be  made  by  rational,  motivated  leaders.  However, 
the  "real  world"  is  that  design  decisions  are  often  made  before  hardware  is 
yet  available  for  pilot  evaluation.  Most  pilot  evaluation  of  new  systems 
is  currently  done  in  the  flight  test  phase  where  the  number  of  design  options 
left  for  evaluation  is  limited.  We  would  gain  a  great  deal  by  accomplishing 
meaningful  evaluations  at  an  early  stage  in  the  design  process.  At  the 
risk  of  alienating  all  of  the  pilots  who  may  read  this,  that  early  evaluation 
means  that  piloted  simulations  need  to  occur  early  in  a  systems  development 
program.  An  early  estimate  of  the  impact  of  various  design  options  will  be 
the  result.  Additionally,  many  more  options  can  be  considered  because 
simulation  is  much  less  expensive  and  because  there  is  more  simulator  time 
available  than  flight  time.  The  present  radar  and  fire  control  computer 
evaluations  in  the  F-16  are  outstanding  examples  of  flight  tests  which  need 
to  be  supplemented  by  good  piloted  simultation  efforts.  Ground  simultation 
of  those  systems  would  allow  changes  to  be  evaluated  much  more  efficiently 
and  allow  the  design  process  to  converge  on  the  best  solution  more  rapidly. 

It  is  mandatory  that  several  pilots  be  involved  in  the  piloted 
evaluation  and  development  of  new  systems  whether  that  is  accomplished 
in  a  simulator  or  on  an  airplane.  During  the  control  law  tests  which  I 
have  managed  on  the  AFTI/F-16  and  on  the  basic  F-16,  the  final  configuration 
as  developed  by  several  pilots  was  never  the  same  configuration  as  would  have 
been  developed  by  a  single  pilot.  In  each  case,  however,  the  final  configura¬ 
tion  was  always  liked  by  all  of  the  pilots.  During  all  of  those  evaluations, 
every  pilot  had  a  VETO  and  every  pilot  had  an  equal  vote  in  the  development 
process.  In  most  cases,  unbiased  pilot  input  was  achieved  by  requiring  the 
pilot  to  write  a  written  report  before  talking  to  other  evaluation  pilots. 

An  optimum  number  of  pilots  apperas  to  be  between  four  and  eight.  Valid 
piloted  simulation  evaluations,  then,  require  several  pilots  participating 
together  in  a  strictly  structured  design  evaluation  process.  Of  course 
the  simulation  inself  and  the  piloting  tasks  involved  must  also  be  realistic. 
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There  is  another  challenge  that  we  at  this  workshop  need  to  accept. 
Pilot-vehicle  interfaces  are  no  longer  limited  by  hardware.  The  limit  can 
be  the  pilot  (workload  saturation)  if  we  don’t  design  good  systems.  Those 
systems  need  to  communicate  with  the  pilot  in  new  and  innovate  ways:  voice, 
color,  etc.  We  need  to  incorporate  state  estimators  to  determine  when  we 
display  what  piece  of  data.  We  need  to  accept  as  fact  the  uniqueness  of 
pilots  which  demands  that  we  include  flexibility  in  the  design  to  accommodate 
individual  differences. 

The  flight  test  community  is  limited  in  many  ways  so  it  is  critical 
that  we  provide  good  pilot-vehicle  analysis  early  in  the  design  process. 

We  are: 

(1)  Time  limited  -  only  so  many  hours  in  the  day. 

(2)  People  limited  -  a  finite  number  of  engineers  and  psycholinguists 
limits  us  in  our  ability  to  look  at  every  possible  new  idea. 

(3)  Dollar  limited  -  we.  cannot  afford  to  flight  test  all  of  the 
different  design  options  available  to  us. 

Because  of  these  real  limitations  we  roust: 

(1)  Study  the  pilot/aircraft  system  at  an  early  point  in  the  design 
process  through  meaningful/realistic  tests. 

(2)  Prioritize  the  data  displayed  to  the  pilot  because  of  the  wide 
range  of  data  available  for  display. 

(3)  Provide  meaningful  workload  and  performance  data  for  decision 
makers  early  in  the  design  process. 

(4)  Develop  systems  which  accommodate  individual  pilot  differences, 

(5)  Develop  systems  which  give  good  total  system  performance. 


APPENDIX 


SKILL  TASK  ACCOMPLISHMENT  RELATED  TO  PILOT  TECHNIQUE  AND  AIRSHOW  SAFETY 

Human  skill  task  performance  needs  to  be  understood  by  all  involved 
in  workload  measurement. 

Skill  tasks  are  learned  by  the  human  through  a  process  which  seems  to 
be  consistent  whether  the  skill  task  is  flying  an  airplane  or  hitting  a 
baseball.  Several  aspects  of  this  process  are: 

(1)  The  mind  works  on  two  levels:  conscious  &  subconscious: 

The  conscious  mind  does  only  one  task  at  a  time  while  the  sub¬ 
conscious  mind  accomplishes  many  tasks  simultaneously.  The  conscious  mind 
time  shares.  Every  event  which  happens  to  you  is  stored  in  the  subconscious. 
Skill  tasks  are  accomplished  best  when  the  task  is  done  at  the  subconscious 
level . 

(2)  Tie  between  Foveal  vision  and  conscious: 

The  strong  tie  between  Foveal  vision  and  the  conscious  mind  is 
important  because  it  establishes  the  necessity  to  have  a  good  visual  cue  in 
each  skill  task:  "watch  the  ball,"  "watch  the  rim,"  "stare  at  the  target." 
Subconscious  hand-eye  coordination  is  developed  in  this  way. 

(3)  Proper  visual  cue  and  practice  is  the  key  to  skill  task  performance: 

The  key  to  skill  task  performance  is  to  use  the  proper  visual  cue(s) 
and  then  to  practice  a  great  deal  to  get  a  good  data  base  in  the  subconscious 
mind  ("program  core") .  Of  all  the  cues  that  we  use  to  accomplish  a  skill 
task,  the  visual  cue  is  by  far  the  most  important.  Probably  90  pet  of  all 
cues  for  doing  skill  tasks  are  visual  cues. 

(4)  Each  pilot  has  a  unique  style: 

It  is  extremely  important  to  understand  that  each  human  pilot  is 
unique.  Every  pilot  flies  differently.  Traditionally,  we  have  tended  to 
imagine  the  standard  pilot  or  the  great  pilot.  He  doesn't  exist.  "The 
world's  greatest  fighter  pilot"  is  found  only  at  the  Nellis  AFB  Officers 
Club  Bar. 

Pilot  performance  of  skill  tasks  in.  flight  very  closely  parallels  that 
of  various  other  skill  tasks:  flying,  tennis,  basketball,  skiing,  riding 
a  bicycle,  etc.  Most  of  the  mental  work  in  these  tasks  is  done  on  a 
subconscious  level.  The  subconscious  work  is  done  in  one  part  of  the  brain 
and  the  conscious  work  is  done  in  another  part.  The  FOVEAL  vision  of  the 
eye  (the  central  vision  or  sharply  focused  vision)  is  processed  by  the 
conscious  part  of  the  brain.  The  subconscious  part  of  the  brain  processes 
both  th$  FOVEAL  and  the  peripheral  vision.  The  conscious  part  of  the  brain 
performs  the  reasoning  or  analytical  functions  and  it  only  does  one  thing 
at  a  time.  In  essence  it  "time  shares"  various  tasks  and  problems  and  takes 
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some  finite  amount  of  time  to  accomplish  each  separate  calculation, 
decision,  or  analysis.  The  subconscious  mind,  on  the  other  hand,  responds 
to  information  and  gives  out  a  large  number  of  commands  simultaneously;  it 
can  do  many  different  functions  at  the  same  time.  With  these  things  in 
mind,  it  is  obvious  that  the  key  to  all  skill  tasks  is  to  do  most  of  the 
work  on  the  subconscious  level.  In  each  separate  skill  task  the  person 
must  use  the  correct  visual  reference  and  then  accomplish  a  large  amount 
of  practice  to  establish  a  large  data  base  in  the  subconscious  mind.  I 
like  to  refer  to  that  practice  process  as  "programming  core"  because  of 
the  obvious  similarity  with  digital  computers.  In  each  separate  skill 
task  there  is  a  significance  to  the  "appropriate  sighting  reference." 

Firstly,  the  conscious  mind  becomes  occupied  when  the  individual  concentrates 
on  that  sighting  reference  and  that  allows  the  subconscious  mind  to  do 
most  of  the  task.  Secondly,  in  every  skill  task  the  mind  needs  some 
visual  reference  as  a  foundation  from  which  to  make  adjustments.  In  flying 
that  visual  reference  is  pitch  attitude.  In  tennis  the  great  (even  good) 
player  watches  the  ball  very  intently.  In  basketball  the  great  player  with 
the  consistent  jump  shot  always  focuses  instinctively  on  some  specific  part 
of  the  basket.  The  great  hitters  in  baseball  watch  the  ball  so  intently 
that  they  can  actually  see  the  spin  of  the  ball.  It  becomes  imperative  in 
all  of  these  tasks  that  the  individual  leave  the  majority  of  the  mental  work 
on  the  subconscious  level;  when  we  try  to  elevate  individual  parts  of  the 
task  to  the  conscious  level  then  we  slow  down  in  our  performance.  An  out¬ 
standing  player  in  any  sport  is  the  individual  who  has  supreme  confidence  in 
himself  and  in  his  coach  so  that  he  leaves  most  of  the  task  on  the  subconscious 
level.  If  the  coach  tells  him  to  do  something  in  a  specific  way  then  he 
practices  that  without  question  until  he  has  "programmed"  the  specific  parts 
of  the  task  on  the  subconscious  level.  When  this  individual  is  asked  how 
he  does  a  task,  he  then  tends  to  answer  by  saying  that  he  does  what  comes 
naturally.  There  are  some  good  players  who  try  to  dig  into  all  the  separate 
parts  of  a  task,  and  by  doing  this,  they  elevate  each  part  of  the  task  to 
the  conscious  level.  They  can  attain  some  high  performance  level  but  they 
never  quite  become  the  greatest.  They  are  just  a  little  bit  slower  than  the 
"natural  athlete"  which  was  described  first.  This  second  individual  becomes 
the  really  great  coach  even  though  he  was  never  the  best  of  his  field  as  a 
player.  This  fact  is  logical  because  the  natural  athlete  never  did  study 
the  intricacies  of  a  task  while  the  good  coach  has  studied  each  task  in 
exhaustive,  thorough,  comprehensive  detail  and  can  therefore  teach  others. 

By  elevating  the  task  to  the  conscious  level  he  understood  it  better,  but 
he  also  prevented  himself  from  being  absolutely  great.  When  a  coach  who 
has  studied  tasks  in  detail  teaches  his  players,  he  speaks  from  a  sound 
foundation  in  fundamentals  and  urges  them  to  master  the  proper  skills.  As 
his  advice  leads  them  to  success,  the  players  gain  faith  in  themselves  and 
their  coach.  They  leave  most  of  the  skill  task  in  the  subconscious  level 
as  they  should  to  gain  great  success. 

You  may  now  ask,  "How  does  this  impact  flying  airplanes?"  Just  as  in 
all  sports,  there  are  parts  of  the  flying  task  which  require  skill  while 
some  things  involve  brute  force.  We  need  to  use  all  of  the  cues  available 
and  we  need  to  have  a  good  visual  cue.  When  we  are  landing  the  airplane  or 
accomplishing  air/air  tracking,  then  we  need  to  practice  those  tasks  several 
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times  to  "program  core."  The  conscous  mind  "time  shares"  its  tasks/ 
decisions;  therefore,  it  becomes  important  to  establish  some  priority 
of  things  that  need  to  be  consciously  monitored  or  checked.  Hence  the 
standard  instrument  cross  check  becomes  very  important  to  flying  good 
instruments.  The  conscious  mind/Foveal  vision  spend  most  of  their  time 
on  the  primary  visual  cue  (ADI) .  Other  parameters  are  reviewed  on  a  time¬ 
sharing  basis.  The  check  becomes: 

A/S  < - ADI  < - >  ALT 

$ 

HSI 

In  addition,  the  engine  parameters,  VVI,  etc,  are  checked  less  frequently. 

The  key  to  A/A  gunnery  is  also  obvious.  Focus  your  eyes  and  conscious  efforts 
on  the  target  primarily.  Let  your  subconscious  mind  do  most  of  the  work. 
"Time-share"  tasks  in  the  correct  sequence:  Radar  lock-on,  get  in  the 
target's  turning  plane,  start  the  pipper  drifting  toward  the  target,  accomplish 
fine  tracking,  shoot  at  the  right  time. 

In  all  of  these  tasks  it  is  very  important  that  the  mind  and  the  body 
be  in  tune.  For  all  of  these  skill  tasks  it  is  extremely  important  that  we 
be  relaxed  but  alert.  Remember,  have  your  inner  self  and  your  outer  self 
in  tune:  Both  your  physical  self  and  your  mental  self  must  be  relaxed  but 
alert. 

There  are  times  when  all  of  these  facts  will  help  you  to  fly  an  air¬ 
plane  better  but  it  will  not  cause  your  death  if  you  forget  them.  However, 
we  periodically  place  ourselves  in  situations  where  the  smallest  mistake 
can  kill  us:  Air/ground  gunnery,  formation,  or  airshows.  In  an  airshow, 
we  are  flying  extremely  close  to  the  ground  and  a  mistake  might  kill  us;  it 
becomes  very  important  to  determine  what  parameters  will  kill  you  and  what 
parameters  will  simply  make  the  show  look  a  little  less  impressive.  For 
each  maneuver  we  establish  a  priority  list  of  parameters  that  need  to  be 
checked.  Most  of  the  flying  task  has  been  given  to  the  subconscious  mind 
while  the  conscious  mind  focuses  on  the  correct  sighting  reference  (aircraft 
attitude  primarily)  and  time-shares  checking  other  parameters  based  on  the 
priority  list  established  before  flight.  If  we  "program  core"  by  practicing 
the  airshow  several  times,  the  airshows  can  be  made  very  safe. 

There  can  be  many  distractions  which  disturb  the  established  routine. 
Anything  that  is  external  to  the  normal  thought  process  may  get  elevated  to 
the  conscious  level  because  "core"  doesn't  know  what  to  do  about  this 
"unprogrammed"  event.  Remember  that  the  conscious  mind  only  does  one 
thing  at  a  time  (it  "time  shares"  separate  tasks)  and  that  the  normal  para¬ 
meter  check  sequence  can  be  broken  by  an  unprogrammed  event.  If  one  of 
those  parameters  that  you  miss  because  of  the  disturbance  is  a  critical  para¬ 
meter,  then  that  disturbance  might  kill  you. 


In  addition  to  external  distractions,  there  are  internal  distractions. 
If  your  mind  is  thinking  about  something  other  than  the  airshow  then  you 
might  miss  a  critical  parameter.  It  is  extremely  important  to  cleanse  the 
mind  of  all  distractions.  Most  pilots  will  do  this  by  going  into  isolation 
for  a  period  of  time  prior  to  their  airshow.  By  mentally  reviewing  each 
maneuver  and  mentally  flying  the  airshow,  then  the  pilot  can  cleanse  his 
mind  of  other  events  which  might  cause  a  fatal  distraction.  All  airshow 
pilots  should  instruct  their  support  crew  and  friends  about  the  effect  of 
a  mental  distraction;  they  should  not  bring  problems  to  the  pilot  when  he 
is  in  his  airshow  preparation.  The  pilot  should  know  the  possible  fatal 
impact  of  mental  distractions  and  should  cleanse  his  mind  of  any  last  minute 
problems  before  he  goes  to  fly  the  airshow.  Any  airshow  pilot  is  "tickling 
the  bear"  if  he  stays  in  the  hospitality  suite  until  just  before  his  flight. 
Such  overconfidence  leads  to  filling  the  conscious  mind  with  unneeded 
thoughts  to  the  conclusion  of  safety-critical  cross  checks.  I  believe  many 
airshow  fatalities  could  have  been  avoided  by  following  this  checklist: 

(1)  Prepare  your  airshow  and  establish  a  parameter  priority  list  for 
each  maneuver. 

(2)  Practice  until  "core"  has  been  adequately  "programmed"  to  give  a 
safe  demonstration. 

(3)  Go  into  isolation  before  the  airshow  to  cleanse  the  mind  of 
extraneous  distractions. 

(4)  Get  your  inner  self  and  your  outer  self  (mental  and  physical)  in 
tune.  You  must  be  relaxed  but  alert. 

(5)  Do  not  allow  external  or  internal  distraction  to  disturb  your 
conscious  mind  in  its  established  parameter  check  sequence. 


ARMY  WORKLOAD  RESEARCH  AND  DEVELOPMENT  REQUIREMENTS 


Richard  S.  Dunn,  PhD 

US  Army  Research  &  Technology  Laboratories 
Ames  Research  Center 
Moffett  Field,  California  94035 


This  article  presents  a  discussion  of  the  principle  current  Army 
requirement  for  research  and  methodology  development  in  pilot  and  aircrew 
workload  measurement.  Emphasis  is  given  to  the  shifting  impact  of  operator 
task  demands  and  to  the  utility  value  of  workload  methodology  in  guiding 
design  and  development  activities.  The  discussion  focuses  on  requirements 
which  are  especially  acute  in  the  system  acquisition  process  involving 
development  of  new  helicopter  crew  stations.  Conclusions  and  opinions  offered 
by  the  writer  suggest  both  a  conceptual  orientation  and  a  general  approach  to 
meeting  the  requirements . 


Improved  workload  measurement  methodology  is  needed  by  the  Army  for 
several  purposes  in  the  helicopter  system  acquisition  process.  Although  there 
may  be  other  potentially  fruitful  applications  for  the  workload  concept,  the 
focus  here  is  on  system  acquisition.  In  many  ways  the  Army  is  in  a  transition 
period  with  respect  to  helicopter  development  and  application.  Both  missions 
and  crew  task  demands  are  undergoing  rapid  and  extensive  change.  Relatively 
recent  changes  include  adoption  of  the  anti-armor  role  as  the  primary  mission 
for  the  attack  and  scout  helicopter  team,  and  the  expansion  of  night  and  all 
weather  capability  to  meet  the  needs  of  continuous  operations.  Extended 
development  of  terrain  flight  tactics  and  long-standoff  target  acquisition 
capability  are  also  sources  of  new  task  components.  In  addition,  mission 
requirements  for  air-to-air  combat  with  helicopters  now  appear  to  be  gaining 
importance. 


Tactical  helicopter  system  designers  are  incorporating  many  advanced 
subsystems  for  crew  job  aids  or  for  automation  of  previously  manual  tasks. 

The  trend  is  already  highly  visible:  disengagement  of  the  pilot  and  crew  from 
routine  aircraft  operation  through  use  of  more  reliable  or  automatic  basic 
subsystems  (powerplant,  transmission,  electrical  supply  etc.)  and  increasing 
task  engagement  with  mission  related  subsystems  (target  acquisition,  weapons 
etc.).  Mission  equipment  packages  are  increasing  in  complexity  and  operator 
attention  demands.  Many  new  subsystem  elements  are  recent  developments  with 
which  the  Army  developers  and  users  have  little  prior  experience.  This  means 
that  dependence  on  convention  or  prior  practice  is  impossible,  and  in  many 
cases  the  task  demands  imposed  on  the  operator  in  system  operation  are 
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unpredictable.  Workload  consequences  are  uncertain  during  development  and  may 
not  be  fully  known  even  after  completion  of  the  developmental  test  cycle. 

At  important  system  development  choice  points,  equipment  tradeoffs  are  made 
with  reliable  cost,  weight,  space  and  development  risk  data  and  only  hesitant 
workload  and  mission  effectiveness  forcasts  or  guesses.  System  project 
managers  cannot  be  blamed  for  often  selecting  the  design  approach  with  lower 
cost,  weight,  size  and  risk  even  though  the  end  result  may  be  untenable 
demands  for  high  performance  on  the  part  of  system  operators. 


Since  poor  system  design  or  excessive  mission  demands  all  result  in  high 
operator  workload  there  has  been  increasing  general  attention  to  the  subject. 
Ironically  we  now  find  ourselves  ready  to  conclude  that  the  workload  idea  has 
little  merit  in  the  scientific  process  of  theory  and  model  building  attempting 
to  account  for  human  behavior.  Because  of  overlapping  meaning  with  numerous 
well  established  behavioral  processes  or  variables,  an  operational  definition 
containing  much  of  the  everyday  sense  of  the  term  has  not  been  forthcoming. 

But  it  is  suggested  here  that  the  popular  appeal  of  the  term  and  renewed 
attention  to  good  human  engineering  in  military  systems  need  not  be  wasted. 
Aside  from  its  potential  explanitory  and  scientific  value,  which  may  be  in 
doubt,  the  workload  idea  may  yet  have  a  high  degree  of  utility  value  when 
developed  for  application  in  the  system  acquisition  process. 


To  reach  this  potential  a  standardized  measurement  methodology  must  be 
developed  and  refined.  One  main  point  of  this  discussion  is  that  the 
methodology  research  should  be  related  in  a  direct  way  to  the  purpose  intended 
for  the  measurement  methods.  An  explicit  statement  of  the  need  and  use  for 
measurement  methods  is  a  key  requirement  in  selecting  an  approach  to  their 
development.  The  conceptual  orientation  suggested  here  is  that  workload 
measurement  methods  should  be  viewed  as  a  product,  the  result  of  a  development 
effort  in  applied  psychology.  The  product  in  this  case  is  for  use  by 
developers  seeking  more  effective  methods  for  helicopter  weapon  system  design, 
development  and  evaluation. 


A  product  oriented  effort  to  tailor  systematic  workload  measurement 
methods  to  a  specific  end  purpose  really  amounts  to  a  psychological  test 
development  task.  It  should  be  noted  that  there  are  both  benefits  and 
drawbacks  to  this  conceptual  orientation.  The  benefits  are  that  developers 
can  start  with  a  careful  evaluation  of  the  requirements  and  work  backwards  (at 
least  in  the  first  stages  of  test  development).  This  is  not  the  normal 
approach  in  research  psychology,  being  a  strategy  usually  available  only  to 
engineering  development.  A  second  benefit  is  that  the  explicit 


techniques  which  will  become  the  standardized  measurement  methodology  probably 
already  exist  in  their  scientifically  oriented  format  along  with  a  body  of 
useful  data.  The  biggest  drawback  to  the  suggested  orientation  is  the  need  to 
recognize  the  limited  nature  of  the  objective.  Scientific  merit  and  universal 
generality  will  (probably)  not  be  the  result.  The  eventual  methods  developed 
may  fall  short  of  the  ideal  for  measurement  tools  and  will  require  careful 
application,  judicious  interpretation  and  some  active  protection  from  misuse. 
Also,  the  numerous  other  fields  in  which  the  workload  idea  may  be  useful 
(personnel  selection,  training,  system  certification  etc.)  may  not  find  the 
tailored  product  applicable  to  their  problems. 


Fully  developed  workload  measures  would  aid  the  system  acquisition 
process  in  many  ways.  To  identify  the  practical  requirements  which  provide 
constraints  or  determining  characteristics  in  the  test  construction  task  a 
discussion  of  the  ultimate  uses  and  benefits  to  be  obtained  is  in  order. 
Workload  measurement  could  provide  a  procedural  basis  and  framework  to 
organize  human  engineering  efforts.  It  would  permit  empirical  comparisons 
between  systems  to  develop  normative  data  reflecting  mission  contingency 
reserve  requirements  and  to  establish  sustainable  workload  levels  for  normal 
mission  operations.  The  same  information  would  aid  in  showing  the  relationship 
between  the  quality  of  system  design  or  operation  and  mission  effectiveness. 
The  methodology  would  permit  systematic  consideration  of  workload  in  all 
design  development  and  evaluation  phases.  By  providing  a  basis  for 
specfication  and  statement  of  work  entries,  it  would  permit  workload  for 
specific  system  functions  to  become  an  engineering  parameter  in  system  design 
tasks.  At  least  for  some  system  functions  workload  would  have  the  status  of  a 
system  characteristic  and  could  be  weighted  fairly  in  tradeoffs  with  other 
parameters  such  as  weight,  speed,  cost,  etc. 


These  end  objectives  imply  a  generality  and  standardization  level  that 
would  permit  the  workload  measures  to  be  applied  to  any  combination  of  system 
or  subsystem  crewstation,  operator,  and  mission  component  or  task.  To  be  most 
useful  in  design,  development  and  evaluation  steps  the  measurement  procedures 
should  characterize  crew  task  demands  which  are  a  function  of  system  design 
quality  and  which  are  under  the  control  of  designers  or  developers.  In 
application,  workload  measures  would  apply  to  specific  tasks  or  functions;  the 
assessment  would  not  be  a  universal  evaluation  of  a  system  for  all  of  its 
potential  uses.  The  workload  measures  need  not  apply  to  workload  determinants 
which  are  outside  the  control  of  system  designers  (except  to  the  degree 
necessary  to  establish  experimental  control  when  required).  Another 
implication  is  that  individual  operators  are  not  the  subject  of  evaluation. 

So  long  as  the  operator's  skills,  motivation,  stress  levels,  risk  taking 
behavior,  operating  strategy,  fatigue  state  and  etc.  are  within  the  range  of 
the  intended  user  population,  these  factors  and  the  individual  differences  so 


strongly  present  within  them  are  not  the  subject  of  the  measurement  procedure. 
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This  overall  requirement,  joint  evaluation  of  equipment  and  tasks  but  not 
individual  operators,  frames  the  basic  test  development  task.  In  addition, 
the  results  should  permit  users  to  focus  on  various  workload  determinants 
associated  with  the  equipment  or  its  operation  in  flexible  ways  depending  on 
the  needs  of  specific  applications.  Workload  measurement  procedures  with 
more  than  one  measurement  or  result  offer  the  possibility  of  a  diagnostic 
type  of  use  in  which  the  task  components  or  equipment  components  associated 
with  workload  issues  could  be  evaluated.  By  relating  the  measurement 
procedures  discussed  here  to  more  rigorously  anchored  behavioral  data  from 
laboratory  research  and  from  subsystem  development  data,  some  capability  to 
predict  workload  for  combinations  of  both  equipment  (more  than  one  subsystem) 
and  tasks  (various  system  functions)  serially  or  together  may  emerge.  Using 
this  capability  the  system  designer  could  control  the  workload  consequences  of 
the  overall  system  design  in  order  to  avoid  peak  overloads,  minimize  errors 
due  to  excess  workload  levels,  control  operator  skill,  training  and 
proficiency  requirements,  establish  balanced  criteria  for  equipment  and 
mission  reliability  and  predict  the  suitability  of  failure  mode  back-  up 
systems. 


Up  to  this  point  the  discussion  has  identified  a  test  construction  task 
with  limited  and  specific  objectives.  The  resulting  procedures  will  not 
reveal  universal  truth  about  workload  for  use  in  theory  or  model  building 
efforts.  They  will  be  a  set  of  operational  definitions  to  establish  the 
procedure  for  and  explicit  meaning  of  workload  measurement.  The  result  may 
not  be  useful  procedures  outside  the  realm  of  system  acquisition.  What  can 
be  hoped  for  is  that  through  construct  validation  the  resulting  measures  may 
eventually  attain  a  cohesive  sensible  meaning  in  their  own  way  that  is 
consistent  with  other  concepts  in  engineering  psychology.  Whether  or  not  this 
occurs  the  measures  developed  may  be  formalized  and  standardized.  They  may 
form  the  basis  of  an  extended  design-development-test  technique  that  augments 
analytical  techniques  such  as  task  analysis  and  is  practical  for  inclusion  in 
system  development  work  statements,  specifications  and  standards.  For  any 
given  procedure,  initial  success  may  be  limited  until  usage  provides 
sufficient  experience  against  which  to  judge  individual  outcomes.  Construct 
validation  requires  converging  lines  of  evidence  from  different  applications. 
At  the  outset,  findings  may  only  generate  rule-of-thumb  guidance  and  surely 
will  not  provide  all  the  potential  benefits  discussed  above  until  fully 
developed  to  a  high  level  of  maturity. 


On  the  way  to  this  perhaps  utopian  objective,  developers  must  encounter 
the  standard  list  of  desirable  properties  for  all  psychological  tests.  The 
test  procedures  and  its  outcomes  should  be  reliable,  efficient,  continous, 
sensitive,  and  non-intrusive.  Only  the  last  of  these  will  be  discussed.  In 
the  present  context  a  non-intrusive  measure  would  not  alter  the  process  in  a 
way  that  would  spoil  the  measurement.  In  workload  research  this  constraint  is 
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usually  taken  to  mean  several  things  that  it  actually  may  not  mean.  Strong 
changes  in  operator  strategy  or  response  style  as  a,  result  of  workload 
measurement  introduction  surely  must  be  avoided.  But  it  may  not  be  possible , 
or  desirable,  to  introduce  workload  measurement  without  any  change  in  the 
operators  task  demands,  without  the  operators  awareness,  or  without  some 
degree  of  conflict  or  overlap  between  the  tasks  underway  originally  and  those 
associated  with  the  workload  measure.  This  is  particularly  true  if  an 
ordinary  secondary  task  approach  is  employed.  In  any  event,  all  that  is 
really  required  to  satisfy  the  non-intrusive  measure  constraint  is  reasonable 
assurance  that  the  process  being  measured  is  not  so  altered  by  the 
measurement,  itself,  that  the  results  are  invalid  for  the  purposes  intended. 


There  are  several  desirable  features  the  writer  would  add  to  the  standard 
list.  First,  the  methodology  should  have  high  face  validity  both  to  test 
subject  system  operators  and  to  the  acquisition  process  managers.  These  are 
user  acceptance  factors  that  are  essential  for  cooperation  and  application  of 
findings.  Also,  specific  techniques  should  be  easy  for  both  evaluators  and 
test  subjects  to  learn.  For  practical  reasons  automatic  administration  and 
scoring  would  also  be  desirable  characteristics  for  any  specific  set  of 
measurement  procedures. 


The  next  consideration  in  reviewing  this  test  construction  task  is  to 
survey  the  known  pitfalls  awaiting  unwary  developers.  For  workload  measure¬ 
ment  there  are  discouragingly  many.  The  most  important  issue  has  already  been 
discussed  and  involves  lack  of  specific  orientation  to  the  application  usage. 
Scientific  efforts  applying  a  reductionistic-analytlc-experimental  approach 
usually  result  in  a  switch  in  topics  from  workload  to  learning,  vigilance, 
motivation,  fatigue,  stress,  individual  differences,  risk  taking,  and  so  on. 
The  level  of  reductionism  attempted  in  a  composite  measure  to  characterize 
workload  must  be  appropriate  to  both  the  setting  of  the  application  and  the 
usage  for  the  results.  Various  practical  problems  in  system  testing  provide 
serious  constraints  which  limit  test  data  observation  opportunities,  and  which 
may  introduce  some  of  the  problems  associated  with  field  testing.  Low 
experimental  control,  unreproducible  scenarios,  varying  external  factors  which 
introduce  situational  conditions  in  system  operation,  and,  at  least  at  the 
outset,  the  absence  of  firm  validation  criteria  must  all  be  considered  by  the 
workload  methods  developer.  Other  problems  or  pitfalls  are  associated  with 
the  optional  timing  of  many  system  operation  tasks,  the  need  to  separate 
workload  from  multiple  system  outputs  -  some  of  which  are  antagonistic  or 
strategy  dependent,  and  the  potential  need  to  deal  with  a  very  large  set  of 
systems  functions  which  may  include  back-up  or  failure  mode  operations.  For 
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many  conditions  of  widely  varying  workload  the  only  difference  in  system 
output  may  be  hazard  or  risk  effects  which  are  unquantified.  In  the  end,  the 
relationship  between  workload  and  system  effectiveness  will  have  to  be  made 
clear  by  empirical  demonstration  or  the  uses  planned  for  the  workload 
information  will  not  emerge. 


For  sure,  there  are  plenty  of  problems  to  overcome  in  the  application  of 
workload  measurement  to  aid  system  development.  But  the  need  is  strongly 
present  and  well  recognized.  The  potential  benefits  are  extensive  and 
important.  Our  behavioral  technology  surely  has  the  necessary  tools  to  attack 
the  problem  in  a  productive  way.  This  convergence  of  motive,  means  and 
meaningful  consequence  for  success  is  all  that  is  really  needed  to  justify  a 
serious  development  effort.  With  the  product  oriented  approach  suggested  here 
a  set  of  procedures  tailored  to  the  end  usage  can  be  developed  that  will 
provide  many,  perhaps  most  or  even  all  of  the  benefits  suggested.  Workload 
methodology  research  can  make  the  strongest  contribution  to  Army  aviation 
system  development  capability  by  meeting  the  challenges  of  this  task. 
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WORKLOAD  REQUIREMENTS  OF  THE 
HELICOPTER  ANTISUBMARINE  WARFARE  MISSION 


by 

LCDR  M.  A.  COSGROVE,  USN 

The  pilot  workload  in  the  helicopter  Antisubmarine  Warfare  (ASW)  environment  is 
unique  in  many  aspects,  particularly  when  coupled  to  the  harsh  environment  of  at-sea 
night  operations.  For  the  purposes  of  this  discussion,  I  will  primarily  address  the  tasks 
involved  in  piloting  the  Navy's  SH-3H  helicopter. 

The  SH-3H  "Sea  King"  helicopter  is  equipped  with  a  mechanical  flight  control  system 
augmented  by  a  hydraulic  boost  system  and  Automatic  Stabilization  Equipment  (ASE). 
Additionally,  it  has  an  "automatic  coupled  approach"  feature  and  a  barometric  attitude 
hold  system.  It  is  similar  to  the  Air  Force's  CH-3E  helicopter,  and  it  is  manned  by  a  crew 
of  four:  a  pilot,  copilot,  and  two  sensor  operators.  Side-by-side  seating  is  provided  for 
the  pilots  in  the  cockpit,  while  the  sensor  operators  man  their  equipment  in  the  cabin 
section  of  the  aircraft. 

The  SH-3H  has  a  maximum  airspeed  of  120  KIAS  with  an  average  mission  endurance 
of  4.5  hours.  It  is  basically  limited  by  a  restricted  power  available  (critical  in  high 
temperature/density  altitude  regimes),  and  it  lacks  radar,  ECM,  and  inertial  navigation 
systems.  The  large  size  of  the  cabin  which  allows  pilots  to  walk  around  during  long 
missions,  its  boat  hull  which  will  allow  water  landings  in  certain  situations,  and  a  new 
tactical  navigation  system  (ASN-123)  contribute  to  the  flexibility  of  the  SH-3  helicopter. 

In  order  to  accomplish  its  primary  ASW  mission,  the  SH-3H  makes  use  of  a  variety  of 
sensors  including  a  dipping  sonar,  Magnetic  Anomaly  Detection  (MAD)  equipment,  and 
various  types  of  active  and  passive  sonobuoys.  It  is  operated  primarily  from  the  decks  of 
the  Navy's  aircraft  carriers  in  day /night  all-weather  conditions;  and  it  is  frequently  flown 
off  the  decks  of  smaller  aviation-capable  ships  such  as  the  USS  SPRUANCE  class 
destroyer  (DD-963).  The  SH-3  is  often  employed  under  Emission-Controlled  (EMCON) 
conditions  without  any  navigation  aids  (i.e.,  TACNAV)  at  ranges  in  excess  of  100  NM  from 
the  carrier.  Although  it  is  primarily  used  in  the  localization  and  attack  role,  it  is  capable 
of  performing  limited  open-ocean  searches.  The  SH-3H  is  also  tasked  with  performing 
surveillance,  search  and  rescue,  and  utility  missions. 

The  pilot  workload  in  the  cockpit  rapidly  builds  up  during  the  following  three  major 
phases  of  the  mission: 

a.  Tactical  scenarios  which  involve  the  use  of  multiple  sensors,  coordination  with 
other  aircraft,  and  multiple  frequency  communication  links. 

b.  Night  sonar  dipping  maneuvers. 

c.  Night  shipboard  approaches  and  landings. 
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During  tactical  scenarios,  the  pilot  is  required  to  devote  a  major  portion  of  his 
concentration  in  tracking  an  enemy  submarine,  while  relying  on  his  past  training  and 
motor  skills  to  maneuver  the  aircraft  to  accomplish  this  objective.  The  pilot  must 
prioritize  his  subtasks  and  time-share  his  focus  of  attention  while  utilizing  up  to  three 
radios  and  internal  communications;  updating  his  position  and  the  positions  of  the 
submarine,  his  wingman,  or  supporting  assets;  coordinating  the  use  of  the  navigation 
system;  and  frequently  referencing  tactical  publications.  All  these  subtasks  must  be 
accomplished  while  safely  flying  the  aircraft  throughout  its  flight  envelope. 

The  night  sonar  dipping  operations  pose  unique  problems  to  the  helicopter  pilot,  which 
are  not  present  in  fixed-wing  operations.  The  pilot  must  fly  a  profile  which  transitions 
the  aircraft  from  a  forward  flight  level  attitude  to  a  zero  ground  speed,  40-foot  hover 
condition,  which  places  the  aircraft  in  a  nose  up,  left  wing  down  attitude.  Although  the 
"automatic  coupler"  is  designed  to  fly  this  profile  with  no  assistance  from  the  pilot,  in 
actuality,  tactical  situations,  weak  Doppler  return  and  weak  ASE  systems  often  require 
the  pilot  to  manually  complete  this  maneuver.  The  difficulty  of  this  maneuver  is  further 
compounded  by  the  lack  of  a  low  airspeed  system,  the  lack  of  outside  visual  cues  during 
IMC  operations,  and  the  effects  of  varying  sea  state  and  wind  velocities.  Departing  the 
hover  again  requires  a  great  deal  of  concentration  by  the  pilot  to  successfully  transition 
from  the  "hover"  attitude  to  a  safe  airspeed  and  altitude  combination  in  order  to  arrive  at 
the  next  calculated  position  of  the  submarine.  Depending  on  the  submarine's  speed  and 
location,  the  time  of  the  evolution  may  range  from  6  to  20  minutes. 

The  final  major  phase  of  the  helicopter  ASW  mission  which  rapidly  builds  up  the  pilot 
workload  is  the  night  shipboard  approach  and  landing.  As  in  the  case  of  the  sonar  dipping 
operation,  the  pilot  must  transition  the  aircraft  from  a  level  forward  flight  condition  to  a 
precise  hover  over  a  moving  deck.  During  the  approach  phase,  he  must  compensate  for 
weak  visual  cues  which  vary  from  ship  to  ship  due  to  a  lack  of  lighting  standardization  of 
the  various  classes  of  aviation-capable  ships.  The  difficulty  of  the  approach  is  further 
compounded  by  the  lack  of  closure  rate  information  available  to  the  pilot.  Unless  the 
pilot  is  flying  a  Carrier-Controlled  Approach  (CCA),  which  is  not  available  on  most  of  the 
aviation-capable  ships,  he  must  rely  on  his  copilot's  vision  to  detect  the  closure  rate  to 
the  ship  under  the  restricted  visual  conditions  mentioned  previously.  (NIGHT 
HELICOPTER  APPROACH  FILM  SHOWN)  When  pilot  has  finally  completed  his  approach 
and  begins  the  landing  phase,  he  is  faced  by  many  additional  harsh  external  factors  which 
control  the  difficulty  of  the  touchdown.  Ship  motion,  varying  winds,  turbulence  from 
ship's  superstructure,  stack  gases,  and  a  confined  landing  area  are  factors  which  demand  a 
high  skill  level  and  compensation  ability  on  the  part  of  the  pilot.  The  relative  motion  cues 
are  also  diminished  at  night,  even  though  the  pilot  is  hovering  directly  over  the  deck. 
(FILM  SHOWING  SH-60B  RETRIEVAL,  ASSISTANCE  TRAVERSE  (RAST)  SYSTEM) 

Throughout  all  of  these  major  workload  phases  of  the  helicopter  ASW  mission,  several 
other  environmental  factors  combine  to  increase  the  fatigue  level  of  the  pilot.  High 
ambient  noise  levels  accompanied  by  varying  vibration  levels  (which  become  severe  during 
the  translational  lift  portions  of  landings  and  takeoffs),  high  cockpit  temperatures  in 
certain  climates,  and  the  multiple  flight  and  equipment  instruments  which  require 
monitoring  have  often  induced  pilot  disorientation  with  sometimes  lethal  results  on  more 
than  one  occasion. 
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Present  technology  offers  many  possible  solutions  in  reducing  the  pilot  workload  in 
this  environment.  The  use  of  a  HUD  or  an  integrated  CRT  should  be  evaluated  to  reduce 
the  pilot's  instrument  scan  requirements  and  disorientation  by  providing  command 
directors  and  an  artificial  horizon.  Closure  rate  equipment  coupled  to  a  low  airspeed 
system  may  be  the  solution  for  problems  encountered  in  the  night  approach  to  landing  and 
sonar  dipping  phases  of  the  ASW  mission.  The  installation  of  inertial  navigation  equipment 
will  reduce  the  inaccuracies  of  present  systems  and  should  limit  the  accompanying  pilot 
anxiety  and  frustration  levels  associated  with  solely  Doppler-based  navigation  systems. 
An  increased  use  of  helicopter  landing  systems,  such  as  the  RAST  system,  should  not  only 
make  the  shipboard  landing  task  much  easier  but  will  allow  the  helicopter  to  be  operated 
in  higher  sea  states  and  weather  conditions. 

The  tasks  of  the  helicopter  ASW  mission  have  been  defined,  but  precise  workload 
measurement  of  these  tasks  is  still  vague  and,  in  some  cases,  unknown.  The  operational 
success  of  the  ASW  helicopter  will  depend  upon  the  correct  identification  of  those  factors 
which  make  the  pilot's  task  difficult,  accurate  measurement  of  pilot  workload  and 
compensation,  and  finally,  evaluation  and  incorporation  of  mission  and  aircraft  equipment 
which  will  improve  the  performance  of  the  helicopter  in  the  ASW  mission. 
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INTRODUCTION 


We  have  never  been  at  a  more  critical  stage  in  the  need  to  be  able 
to  measure  workload  and  performance  in  both  the  cockpit  and  air 
traffic  control.  In  order  to  illustrate  fully  why  this  is  so,  I 
would  like  to  describe  briefly  the  Federal  Aviation  Administration's 
(FAA)  engineering  and  development  programs  in  Human  Factors. 

The  human  factors  area  has  been  the  subject  of  much  study  over  many 
years  and  the  results  of  these  studies  have  had  a  major  impact  on 
the  aircraft  and  air  traffic  control  systems  in  use  today.  In 
conducting  our  current  efforts  we  are  not  starting  from  "scratch," 
but  building  on  and  improving  the  already  high  performance  of  the 
current  system.  The  focus  of  the  current  FAA  efforts  is  not  on 
"knobology"  or  the  location  of  displays  and  controls  best  suited  to 
the  physiology  of  the  human  being,  although  this  is  certainly  an 
important  area.  Rather,  it  rests  on  areas  such  as  the  following: 

1.  The  causes  and  types  of  human  error  and  the  impact  of  these 
errors  on  the  safety,  performance  and  productivity  of  aircraft 
and  air  traffic  control  system  operations; 

2.  The  definition  of  automation  approaches  that  assume  the  continued 
existence  of  human  as  well  as  machine  error  and  strive  to  reduce 
both  the  occurrence  and  the  consequences  of  such  error; 
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3.  Assessment  of  the  proper  distribution  of  air  traffic  and  air¬ 
craft  control  monitoring  functions  between  automation  systems, 
the  controller  and  the  pilot; 

4.  Determination  of  the  appropriate  interfaces  between  the  man  and 
the  machine  at  each  step  up  the  ladder  leading  to  higher  levels 
of  automation;  and 

5.  Determination  of  adequate  automated,  semi  -  automated  and  manual 
system  backup  capabilities  to  permit  safe  continuation  of  system 
operations  under  a  variety  of  conditions  of  human  and  machine 
system  failures. 

These  areas  of  R&D  are  all  directed  toward  the  need  to  maintain  and 
enhance  the  safety  of  the  aviation  system,  to  achieve  improved 
performance  of  the  system  for  the  participants  and  the  flying 
public,  and  to  make  the  system  more  productive  while  constraining 
the  cost  of  the  system  to  the  Nation. 

Before  proceeding  with  an  overview  of  the  FAA's  human  factors 
program,  I  would  like  to  give  you  a  short  description  of  what  the 
Air  Traffic  Control  (ATC)  system  of  the  year  2000  might  look  like 
including  both  ground  automation  and  airborne  system  capabilities. 
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SCENARIO  FOR  THE  FUTURE  SYSTEM 


The  air  traffic  control  system  in  the  year  2000  will  continue  to  be 
a  civil / m i 1 i t a r y  integrated  system  and  will  continue  to  grow  in  an 
evolutionary  manner.  In  the  future,  in  some  airspace,  services  will 
be  provided  by  an  essentially  fully  automatic  ground-based  system, 
coupled  with  airborne  automated  systems.  On  the  other  hand,  there 
will  still  be  airspace  where  no  air  traffic  control  ground-based 
services  will  be  provided,  and  none  needed.  We  s°e  the  possibility 
of  placing  more  responsibility  into  the  cockpit,  thus  changing  the 
balance  of  control  responsibility  between  pilots  and  controllers. 

Where  there  is  full  participation  in  ATC  services  and  in  major 
terminal  areas,  the  highest  level  of  automation  will  be  employed  to 
support  the  separation  service.  The  basic  control  system  will  be 
designed  to  offer  very  high  reliability  with  reversion  capability  to 
a  safe  backup  automatic  control  system.  Reliability  of  the  automated 
system  will  be  achieved  by  top-down  redundancy  at  individual  control 
facilities,  and  f aci 1 i ty-to-f ac i 1 i ty  backup  using  a  flexible  data 
communications  network.  The  automated  system  will  be  monitored 
automatically  also  by  an  outer-loop  backup  advisory  system  which 
will  independently  provide  information  directly  to  the  cockpit  and 
to  controllers.  Another  level  of  protection,  which  may  also  serve 
to  build  confidence  in  automation,  may  be  achieved  by  use  of  cockpit 
traffic  displays  utilizing  the  data  link  information  from  the  ground 


32 


and/or  an  independent  collision  avoidance  system.  Flexible  use  of 
area  navigation  and  an  automated  ground-based  system  capable  of 
handling  large  numbers  of  variables  will  allow  a  high  degree  of 
routing  freedom,  both  laterally  and  vertically.  Our  work  on  the 
Automatic  En  Route  ATC  system,  called  AERA,  will  serve  as  the  basis 
for  this  system.  We  believe  that  even  in  this  airspace,  some  degree 
of  self-separati on  may  be  possible  if  traffic  information  can  be 
provided  to  the  cockpit  in  such  a  way  that  aircraft  are  aware  moment- 
to-moment  of  the  position  and  perhaps  the  intent  of  other  aircraft, 
and  that  procedures  can  be  worked  out  to  assure  that  pilot  and 
controller  actions  are  fully  understood  and  coordinated. 

In  the  second  level  of  service,  provided  in  low-  and  medium-density 
traffic  areas  with  mixed  IFR/VFR  operations,  the  service  will  be 
similar  to  the  present  system,  but  with  automated  tools  such  as 
metering,  sequencing  and  spacing,  conflict  alert,  and  conflict 
resolution  advisories  assisting  the  controller. 

In  the  third  level  of  service,  IFR  operations  under  Instrument 
Meteorological  Conditions  (IMC)  will  be  permitted  on  the  basis  of 
procedural  rules  in  a  manner  similar  to  that  currently  employed 
under  non-radar  procedures.  This  service  will  be  available  to 
provide  some  degree  of  control  to  protect  aircraft  in  areas  where 
there  is  a  limited  number  of  aircraft  on  IFR  flight  plans. 
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We  have  defined  a  series  of  capabilities  for  the  ATC  system  at  the 
turn  of  the  century  which  will,  in  turn,  require  the  development  and 
implementation  of  services  and  tools  beyond  those  available  today. 

Let  me  mention  a  few  of  those  services. 

The  system  today  provides  a  series  of  services  including  flight  plan 
and  radar  data  processing,  semi-automatic  hand-off  between  sectors, 
automatic  i nter-comouter  communications,  conflict  alert,  terminal 
minimum  safe  altitude  warning,  central  flow  control,  fuel  advisory 
departure  service,  a  central  military  flight  reservation  service. 
Flight  Service  Station  services  to  the  general  aviation  community, 
and  a  whole  range  of  airport  ATC  services. 

For  the  future  we  see  a  system  which  will  have  the  capability  of 
essentially  al'l -weather  .'operations ,  supported  by  Microwave  Landing 
Systems  (MLS);  en  route  minimum  safe  altitude  warning;  a  dramatically 
improved  weather  service  for  the  en  route  system,  major  terminals, 
and  low  activity  terminals;  automated  Flight  Service  Station 
services;  conflict  resolution  advisories;  and  Automatic  Traffic 
Advisory  Services.  The  automation  system  will  be  extensive  and  will 
include  an  automated  en  route  metering  system  and  conflict  probe,  a 
national  integrated  flow  management  service  for  en  route,  terminal 
and  airport  operations,  conflict-free  automatic  fuel  optimal  direct 
routings,  and  fully  automatic  hand-off. 


As  additional  services  and  tools,  we  expect  to  see  an  automatic  air/ 
ground  communications  service  which  will  serve  a  variety  of 
functions.  We  expect  to  have  a  more  nearly  automatic  airport 
surface  surveillance  and  control  capability  at  major  terminals,  an 
automatic  digital  data  weather  distribution  system  which  will  draw 
on  a  new  National  weather  network,  and  the  capability  to  achieve 
reduced  vertical  separation  above  FL  290. 

A  Threat  Alert  and  Collision  Avoidance  System  (TCAS)  will  be 
available,  based  on  the  existing  and  improved  Secondary  Surveil¬ 
lance  Radar  System  (called  Mode  S  by  the  International  Civil 
Aviation  Organization,  ICAO).  It  will  render  valuable  service 
especially  in  over  ocean  areas  and  in  areas  where  ATC  services  are 
limited. 

We  expect  to  see  progress  in  permitting  general  aviation  aircraft  to 
operate  predominantly  outside  the  ATC  system,  if  they  choose  to  do 
so,  while  retaining  the  protection  of  the  system  where  collision 
risk  is  significant. 

Our  work  continues  in  establishing  the  optimum  role  for  cockpit 
displays  of  traffic  information,  and  the  problems  of  wake  vortex 
alleviation  remain  a  major  challenge. 

The  cockpit  of  the  future  will  be  different  from  today's,  and  we 
will  arrive  there  by  an  evolutionary  process.  A  number  of  changes 
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are  already  visible  in  the  laboratories  of  industry,  NASA  and  FAA 
and,  of  course,  the  military.  The  key  word  is  undoubtedly  inte¬ 
gration-  -  i  ntegr  at  i  on  of  displays,  integration  of  functions,  and 
integration  of  warnings  and  alerts.  Highly  flexible  cathode  ray 
tube  or  plasma  displays  are  already  appearing  and,  when  proven  fully 
reliable,  will  undoubtedly  be  highly  valuable  and  flexible  tools  for 
the  information  exchange  between  the  pilot,  his  aircraft  and  the 
ground  system.  We  will  see  these  new  display  systems  integrating 
not  only  the  information  currently  available  in  cockpits,  but  also 
new  information  from  a  variety  of  sources.  Among  these  are  the 
performance  management  computers,  already  demonstrating  the  capa¬ 
bility  of  providing  major  fuel  savings.  They  will,  in  most 
instances,  be  integrated  with  3D  or  4D  area  navigation  capability. 

New  information  will  also  appear.  There  will  be  better  displays  of 
wind  shear  information,  data  link  terminals  which  will  connect  the 
cockpit  to  the  company  and  to  the  air  traffic  control  automation 
system  via  digital  commun i cat i ons  . 

Threat  alerts  on  nearby  aircraft  and  collision  avoidance  information 
will  be  displayed.  Integrating  displays  which  will  simplify  the 
execution  of  curved  or  segmented  paths  made  possible  by  MLS  will  be 
in  the  new  cockpit.  We  will  see  a  major  trend  to  a  master  navigation 
computer  with  multiple  sensor  inputs  from  such  systems  as  INS,  OMEGA, 
VOR/DME,  DME/DME,  MLS  and  perhaps  GPS.  It  is  possible  that  full-time 
displays  of  nearby  traffic  of  concern  will  find  their  way  into  air¬ 
line  and  perhaps  other  cockpits,  based  on  either  air-derived  or 
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ground-derived  information.  They  may  be  used  for  a  variety  of 
purposes  ranging  from  simple  monitoring  of  the  performance  of  an 
automated  ground  system,  to  confidence-building  for  aircraft  on 
nearby  parallel  approach  paths,  to  potentially  more  sophisticated 
capabilities  such  as  sequencing,  station-keeping,  etc. 

The  1990  era  transport  cockpit  will  likely  be  an  all  digital,  all 
electronic  flight  deck  with  high  resolution  multifunction  displays 
that  will  monitor  flight  operations. 

Automated  systems  and  electronic  displays  will  simplify  aircraft 
procedures  in  the  NAS.  To  streamline  the  control  process,  both 
ground-based  and  air-derived  surveillance  information,  meteorolo¬ 
gical  data,  knowledge  of  intent,  and  other  information  will  be 
exchanges  between  automatic  airborne  and  ground  systems.  Both  pilot 
and  controller  will  monitor  and  assess  system  performance,  becoming 
managers  of  highly  automated  systems.  The  NAS  will  offer  high 
reliability  and  yet  retain  reversion  capability  to  a  backup  manual 
control  system.  New  systems  in  the  future  cockpits  will  include 
aircraft  automatic  control,  fly-by-wire,  flight  management  systems, 
advanced  navigation  systems,  integrated  multifunction  displays,  and 
digital  avionics  controls. 

The  high  end  of  the  General  Aviation  spectrum  of  aircraft  will  no 
doubt  see  the  incorporation  of  multifunction  CRT  displays,  flight 
management  systems,  advanced  navigation  and  communications  systems 
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including  data  link  capability.  Cockpits  will  also  include  coupled 
autopilots,  auto  throttles,  and  an  integrated  EHSI  to  include  a 
navigation  display  option  overlaid  with  color  radar  weather  data. 
Features- s*uch  as  on  screen  touch  control,  head  up  display,  inter¬ 
active  voice  response  system  and  direct  access  communications 
between  airborne  and  ground  computers  are  currently  in  various 
stages  of  development. 

On  the  low  end  of  the  spectrum,  the  degree  of  sophistication  will  be 
highly  dependent  on  the  owner's  economic  capability  and  operational 
requirements,  not  on  the  technology  required  to  perform  the  func¬ 
tions.  An  experimental  system  now  in  the  early  stages  of 
development  within  the  FAA  will  provide  the  small  aircraft  with  a 
real-time  ground  weather  radar  picture  in  the  cockpit.  This  uplink 
capability  can  also  provide  NOTAM  information  as  well  as  SIGMETS, 
AIRMETS,  PIREPS,  area  weather  forecasts,  hourly  sequency  reports  and 
ATC  information.  As  with  all  other  systems,  the  display  capability 
will  be  based  solely  on  the  economic  ability  of  the  owner.  It  is 
estimated  that  a  low  cost,  perhaps  as  little  as  $500,  CRT  display 
could'be  developed. 

BACKGROUND 


In  1975,  a  special  DOT  task  force  study  in  the  FAA  safety  mission 
recommended  that  "FAA  undertake  a  major  safety  research  program  to 
assure  that  future  systems  are  designed  around  reasonable  criteria 
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for  human  error."  Consequently,  the  FAA  Office  of  Systems 
Engineering  Management  undertook  a  study  to  identify  human  factors 
problems  associated  with  both  air  carrier  and  general  aviation 
accidents  and  incidents.  This  FAA  study  entitled,  "Program  for 
Optimizing  Crew  Performance  and  Minimizing  Human  Error  in  Aircraft 
Cockpits  (Response  to  DOT  Safety  Recommendation  10)"  used  as 
inputs  safety  statistics  from  a  variety  of  sources  and  solicited 
the  views  of  the  aviation  community  for  its  perception  of  human 
factors  problems  and  potential  solutions.  After  a  great  deal  of 
internal  and  external  discussion,  several  major  problem  areas  were 
identified  as  primary  candidates  for  expanded  effort  and  formed 
the  basis  for  establishing  our  human  factors  program. 

While  research  and  development  in  human  factors  had  been  carried 
on  for  many  years  in  association  with  specific  projects,  FAA 
determined  in  1977  that  a  common  thread  existed  between  the 
programs  and  problems  and  that  central  management  was  needed  to 
assure  a  fully  cohesive  program  which  responded  to  identified 
problems.  The  programs  were  grouped  into  two  broad  areas  related 
to  pilot  and  controller  problems.  Because  even  the  term  "human 
factors"  is  frequently  misunderstood,  we  chose  to  talk  about  our 

programs  in  terms  of  the  intended  result;  namely.  Aircrew  j 

« 

I 

Performance  Enhancement  and  Error  Reduction  (APEER)  and  Controller 

.  i 
.  | 

Performance  Enhancement  and  Error  Reduction  (CPEER). 
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Today,  I  would  like  to  give  a  brief  overview  of  some  of  the  efforts 
we  have  underway  on  the  aircrew  side.  Many  of  these  programs 
represent  joint  efforts  between  the  FAA,  NASA,  and  the  Department  of 
Def ense whi ch  were  undertaken  to  assure  that  the  Nation's  best 
resources  are  applied  efficiently  to  this  problem.  J 

AIRCRAFT  COCKPIT  AND  AIRCREW  HUMAN  FACTORS  ACTIVITIES 

Our  program  in  the  aircraft  cockpit  and  aircrew  area  consists  of 

■ 

several  types  of  activity  including  Human  Factors  Safety  Analysis, 
Aircraft  Certification,  Airman  Certification,  and  Aircrew/ATC 
I nteracti on . 

HUMAN  FACTORS  SAFETY.  In  the  area  of  safety  analysis,  we  have' 
established  activities  designed  to  quantify  the  problems  and 
identify  needed  engineering  and  development  activities. 

o  Pilot  Error  Analysis.  Historically,  pilot  error  is  cited  as  a 
factor  in  approximately  60  percent  of  air  carrier  and  88  percent 
of  general  aviation  fatal  accidents.  Pilot  error  is  also  cited 
as  a  significant  factor  in  aviation  incidents.  A  continuing 
study  is  being  made  of  the  types  and  causes  of  human  error  to 
establish  a  basis  for  improvement  of  current  systems  and  design 
of  new  systems. 

o  We  have  underway  a  general  aviation  accident  problem  analysis. 

A  detailed  categorization  of  accidents  attributed  to  pilot  error 
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and  the  identification  of  underlying  human  factors  problems  was 
needed.  This  was  accomplished  through  a  review  of  general 

aviation  accident  and  incident  data  bases  to  determine  and 

* 

prioritize  human  factors  problem  areas.  A  significant  part  of 
this  work  included  an  examination  of  the  relationships  between 
weather-related  accidents  and  current  methods  of  instrument 
flight  training.  The  program  will  also  define  and  examine  the 
effectiveness  of  alternative  programs. 

o  Commuter/Air  Taxi  Human  Factors  Problems  Analysis.  Commuters 
and  air  taxis  have  the  highest  accident  rate  in  commercial 
aviation,  and  our  objective  in  this  problems  analysis  is  to 
provide  the  FAA  operating  services,  the  industry,  and  the 
engineering  and  development  community  with  a  better  appreciation 
of  the  causes  and  priorities  of  human  error  problems  peculiar  to 
commuter  operations. 

AIRCREW/ATC  INTERACTION.  The  addition  of  new  systems,  mentioned 
above,  into  the  cockpit  and  ATC  system  has  a  complex  effect  on  the 
controller/aircrew  interface.  We  have  developed  several  programs 
specifically  intended  to  determine  the  liabilities  and  benefits  of 
various  methods  of  implementing  advanced  systems. 

o  Cockpit  Data  Information  Requirements  and  Analysis.  The 
* 

introduction  of  advanced  cockpit  design  concepts  and  advanced 
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ATC  system  improvements  will  present  new  requirements  for  cock¬ 
pit  information  processing  and  display.  It  is  essential  that 
human  and  aircraft  system  capabilities  work  in  harmony  with  the 
evolving  ATC  system.  We  plan  to  develop  a  series  of  recommen¬ 
dations  for  efficient  means  of  displaying  and  using  information 
in  the  cockpit,  for  consolidation  of  information  on  electronic 
displays  and  for  functional  integration  of  aircraft  systems. 
Proper  integration  of  such  new  capabilities  as  collision 
avoidance  advisories,  wind  shear  information.  Microwave  Landing 
System  flexible  approach  paths.  Cockpit  Displays  of  Traffic 
Information,  flight  management  computers,  and  others,  is 
essential.  A  similar  review  of  information  requirements  is 
planned  for  the  helicopter  area. 

Separation/Navigation  Standards  for  En  Route  Operations.  Since 
flight  technical  error  of  various  kinds  has  a  major  impact  on 
acceptable  separation  standards,  this  effort  is  intended  to 
develop  improved  methods  of  analysis  to  guide  determination  of 
future  separation  standards  with  particular  emphasis  on  the 
reduction  of  flight  technical  navigation  errors  of  various 
causes.  Our  present  program  is  examining  the  relationship 
between  separation  standards  and  navigation  system  performance 
for  en  route  operations.  Human  error  and  blunders  in  navigation 
are  signficant  contributors  to  the  failure  of  aircraft  to 
navigate  within  designated  routes.  The  program  addresses  the 
human  factors  problems  related  to  the  use  of  current  VOR  and 


42 


area  navigation  systems  which  may  contribute  to  the  error  and 
blunder  problem.  Important  objectives  are  to  examine  advanced 
navigation  system  concepts  and  to  establish  the  data  base  needed 
to  define  guidelines  and  criteria  that  will  recognize  the 
special  needs  of  single  pilot  IFR  operations. 

o  Cockpit  Display  of  Traffic  Information  (CPU) .  While  the 

technology  to  provide  traffic  information  in  the  cockpit  exists, 
the  pilots  ability  to  use  this  information  and  the  impact  this 
will  have  on  the  ATC  system  is  not  fully  known.  Our  objective 
is  to  evaluate  the  use  of  Cockpit  Display  of  Traffic  Infor¬ 
mation  for  both  passive  monitoring  and  active  spacing  tasks  so 
that  the  advantages  and  disadvantages  of  such  use  can  be 
measured  in  terms  of  system  safety,  capacity,  and  efficiency  in 
operationally  realistic  environments.  We  want  to  evaluate  the 
impact  of  CDTI  on  the  pilot  and  on  the  controller,  as  well  as 
the  impact  of  CDTI  on  traffic  flow  stability,  dynamic  merging 
and  spacing,  display  content  and  format,  and  pilot/controller 
workload  changes.  Closely  related  to  this  work  are  efforts  to 
develop  and  evaluate  optimal  displays  for  the  Traffic  Alert  and 
Collision  Avoidance  System  (TCAS)  and  the  Automatic  Traffic 
Advisory  Service  (ATAS). 
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AIRMAN  CERTIFICATION.  These  projects  were  developed  in  direct 
response  to  specific  Aviation  Standards  requests  for  the  particu¬ 
lar  research  to  be  accomplished.  They  include  investigations  of 
training  programs  for  private,  instrument  and  multiengine  ratings 
and  proficiency  training  programs  with  emphasis  on  low  cost 
mechanisms  and  the  value  of  improving  decisionmaking  training. 

o  Private  Pilot  Certification.  Since  the  cost  and  complexity  of 
operating  aircraft  is  increasing  dramatically,  this  effort  is  to 
determine  optimum  training  and  proficiency  check  techniques  for 
private  pilot  certification. 

o  Low-Cost  Proficiency  Training.  In  a  related  effort,  since  the 
cost  of  flying  has  risen  dramatically,  we  are  looking  at 
low-cost  methods  to  train  and  maintain  pilot  proficiency  by 
increased  use  of  simulation,  although  we  are  aware  that  there 
are  real  limitations  in  extensive  use  of  simulators. 

o  Instrument  Rating  Study.  Since  most  weather-related  accidents 
occur  with  VFR  pilots  having  less  than  200  hours  total  time, 
this  work  is  intended  to  shed  light  on  the  feasibility  of 
granting  an  instrument  rating  certification  prior  to  reaching 
200  hours  total  time. 


o  We  are  conducting  a  study  on  the  relationship  of  general 

aviation  pilot  judgment  and  training  to  aircraft  accidents. 
Inappropriate  judgment  is  suspected  of  being  a  prime  cause  of 
pilot  error  in  general  aviation  accidents.  Our  objective  is  to 
develop  a  system  of  experiments  to  assess  pilot  judgment  in 
selecting  appropriate  actions  under  varying  cockpit,  ATC  and 
aircraft  emergency-conditions.  We  plan  to  examine  the  feasi¬ 
bility  of  developing  training  for  improved  decisionmaking  to 
determine  if  pilot  judgment  training  can  offer  specific 
benefits.  This  program  will  result  in  development  of  a  syllabus 
which  will  help  teach  proper  decisionmaking,  defined  as  the 
optimal  application  of  learned  facts,  and  validation  of  this 
concepts  through  field  tests. 

HUMAN  FACTORS  GUIDELINES  FOR  AIRCRAFT  CERTIFICATION.  The  majority 
of  our  aircrew  human  factors  efforts  deal  with  the  introduction  of 
new  systems  and  procedures  into  the  cockpit  and  our  support  for  the 
operating  services  in  their  certification  processes.  This  includes 
the  review  of  current  regulations  and  procedures  related  to  the 
human  factors  area  with  a  view  toward  identifying  potential  changes 
related  to  desired  system  improvements. 

o  FAR  Part  25  Cockpit  Standardization.  Transport  aircraft  have 
been  examined  with  regard  to  cockpit  standardization,  to 
identify  the  potential  problems  that  may  relate  to  equipment 
non-standardization.  We  have  conducted  a  survey  of  seven 
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representative  airlines  to  determine  the  present  status  of 
cockpit  standardization  between  aircraft  of  the  same  type  and 
between  aircraft  of  different  types  as  an  a.d  to  identifying  .any 
problems  associated  with  non-s t andard i z at i on .  The  product  of 
this  work  entitled,  "Transport  Aircraft  Cockpit  Standardization" 
reports  information  on  the  current  status  of  standardization  and 
the  benefits  of  additional  standardization  and  is  currently 
being  reviewed  for  comment  by  the  SAE  S7  Committee  to  assess  the 
possible  value  of  additional  standards  for  comm ere i al  aircraft 
cockpits. 

Altitude  Callouts  During  ASR  Approaches.  A  program  has  recently 
been  completed  on  the  effect  on  pilot  performance  of  controller 
altitude  callouts  during  Airport  Surveillance  Radar  (ASR) 
approaches.  This  program  addressed  the  value  of  providing 
mandatory  altitude  callouts  by  controllers  curing  ASR  approaches 
in  reducing  landing  accidents.  The  conclusion  of  this  work 
indicated  that  the  presence  of  altitude  callouts  did  not 
significantly  affect  pilot  performance  in  executing  ASR 
approaches . 

Aircraft  Alerting  and  Warning  Systems.  Current  transport 
aircraft  systems  are  being  examined  to  determine  those  factors 
which  could  contribute  to  pilot  judgment  error  and  incorrect 
remedial  actions.  Further,  current  systems  may  not  indicate  the 
optimal  order  in  which  critical  actions  should  be  taken  when 
multiple  or  catastrophic  failures  occur. 
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This  program  has  been  underway  for  several  years,  with  parti¬ 
cipation  from  the  three  major  U.S.  civil  transport  aircraft 
manufacturers.  Our  objective  is  to  develop  guidance  for  the 
functional  standardization  of  air  transport  cockpit  alerting 
systems,  particularly  with  regard  to  the  use  of  automation  and 
new  displays  of  alerting  and  warning  data.  We  have  encouraged 
the  airframe  manufacturers  to  work  together  to  coordinate  the 
development  of  a  standardized  industry  alerting  system  concept. 

A  major  study  entitled,  "Aircraft  Alerting  Systems  Standardi¬ 
zation  Study"  has  been  completed  which  lays  out  the  dimensions 
of  the  problem,  and  recommends  specific  alerting  system  concepts 
which  were  tested  in  simulation.  We  are  planning  to  go  beyond 
this  effort  to  concentrate  on  more  advanced  methods  of  warning 
which  take  into  account  the  changing  priorities  for  warnings 
withflightphase. 

o  Computer  Aided  Decision  Making.  We  have  been  working  on 

research  into  more  intelligent  warning  systems  which  can  provide 
not  only  prioritized  alerts  and  warnings,  but  which  may  also  be 
able  to  provide  diagnostic  capabilities  that  will  offer  the 
pilot  the  best  alternative  course  of  action  instantly,  based  on 
computer-aided  analysis  of  the  aircraft  state.  Current  air 
carrier  aircraft  have  complex  emergency/failure  procedures  and 
checklists,  and  in  cases  of  multiple  system  failures,  the 
likelihood  of  intermingling  checklist  procedures  is  high  and 
the  consequences  potentially  severe.  In  this  program  we  have 
investigated  the  feasibility  of  applying  computer-aided 
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decisionmaking  to  analyze  complex  and  interacting  aircraft 
systems  so  that  unusual  failure  situations  can  be  detected  and 
remedial  actions  recommended  to  the  pilot.  We  believe  this  work 
may  show  that  computers  having  a  knowledge  data  base  and 
programmed  reasoning  ability  can  assist  the  pilot  in  high 
workload  situations. 

o  Development  and  evaluation  of  Head-Up  Display  presentations  for 
civil  aviation  aircraft  has  been  undertaken.  The  program  seeks 
to  define  alternative  display  presentations  and  assess  the 
potential  benefits  and  also  any  liabilities  of  this  type  of 
information  presentation,  in  contributing  to  safer  operations  in 
air  carrier  aircraft  during  approach  and  landing.  FAA  has 
established  a  joint  program  with  NASA  to  examine  the  potential 
of  Head-Up  Displays  to  aid  the  flight  crew  in  reducing  pilot 
workload,  increasing  reliability,  and  providing  redundancy  of 
information  for  navigation,  flight  path  control,  and  other 
flight  management  tasks.  The  performance  of  flight  crews  using 
the  device  will  be  assessed  over  a  full  range  of  operational  and 
weather  scenarios.  Our  purpose  is  to  provide  enough  basic  data 
to  the  industry  and  to  FAA's  Aviation  Standards  organization  to 
establish  the  capabilities,  limitations,  and  minimum  require¬ 
ments  for  such  systems. 
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o  Our  work  on  the  wind  shear  program,  which  is  essentially 

complete,  included  a  great  deal  of  emphasis  on  the  human  factors 
aspects  of  the  problem;  namely,  how  best  to  determine  and 
present  the  information  to  the  pilot.  The  airbone  wind  shear 
program  began  with  a  series  of  manned  flight  simulation  experi¬ 
ments  to  identify  and  then  refine  the  most  effective  pilot 
aiding  concepts.  Most  subject  pilots  favored  a  system  that 
displayed  an  airspeed-ground  speed  comparison.  Another  system 
that  rated  well  in  the  evaluation  utilized  a  "quickened"  flight 
director  logic.  These  results  were  validated  in  a  number  of 
simulations  with  airline  and  FAA  pilots,  and  the  results  have 
been  made  available  to  the  industry.  The  current  effort  is  to 
determine  and  validate  a  wind  shear  model  for  use  as  a  standard 
in  simul ator  trai ni ng  . 

o  Alternative  Collision  Avoidance  System  Displays.  Since  threat 
alert  and  collision  avoidance  systems  are  new,  this  effort  is 
intended  to  determi ne  'the  minimum  display  requirements  and  to 
determine  the  optimum  display  characteristics  and  information 
requirements  for  such  systems. 

o  General  Aviation  Weather  Products.  This  effort  is  intended  to 
demonstrate  the  utility  of  transmitting  and  displaying  the 
low-cost  weather  information  to  light  general  aviation  aircraft 
in  order  to  help  reduce  weather-related  accidents  and  incidents 
to  non-radar  equipped  aircraft. 
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In  addition  to  the  programs  mentioned  above  which  have 
applications  to  helicopters,  we  also  have  underway  or  are 
planning  a  number  of  programs  that  relate  specifically  to  the 
human  factors  problems  associated  with  helicopters.  One  of 
these  programs  is  designed  to  define  the  minimum  acceptable 
handling  qualities  for  IFR  flight  in  helicopters.  Other  efforts 
include  analysis  of  accident  data  and  a  survey  of  helicopter 
operators  to  identify  potential  helicopter  problems  and 
characteristics  which  may  contribute  to  helicopter  accidents. 

o  Pilot  Workload  Measures.  Our  effort  here  is  to  develop  and 
validate  workload  measures  to  assist  in  certification  of 
aircraft  and  new  equipment.  Although  a  great  deal  of  work  has 
been  done  on  the  subject  of  defining  pilot  workload  measures, 
additional  efforts  are  needed  to  develop  fully  acceptable, 
scientifically  validated  and  widely  accepted  methods  for 
measuring  pilot  workload.  Some  of  the  current  efforts  underway 
to  deal  with  this  problem  include: 

1.  Completion  of  a  report  entitled,  "Flight  Crewmember  Workload 
Evaluation"  covering  workload  measurement  techniques  that 
have  contributed  to  success ?>j  1  air  transport  certification 
programs . 

2.  A  joint  activity  with  United  States  Air  Force  to  survey  and 
categorize  all  existing  or  planned  workload  assessment  and 
measurement  techniques. 
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3.  An  effort  to  develop  and  validate  a  set  of  subjective  pilot 
workload  measures  that  can  be  used  to  assess  reliably  the 
workload  associated  with  advanced  cockpits  of  aircraft 
op-erating  in  current  and  future  ATC  systems.  The  intended 
end  product  will  be  a  set  of  pilot  rating  scales  for  total 
workload  measuement  which  is  widely  accepted  and  which  can  be 
used  by  government  and  industry  researchers  as  a  common 
measurement  standard.  As  an  initial  activity  in  this 
program,  the  subjective  workload  rating  scale  developed  by 
MIT,  which  is  based  on  an  earlier  method  developed  by 
Cooper  and  Harper  of  NASA,  is  currently  being  examined  and 
validated  at  the  FAA  Technical  Center  as  part  of  the 
"Technical  Assessment  of  Pilot  Effectiveness  (TAPE)"  program. 
John  Fabry  from  the  Technical  Center  will  present  his  recent 
research  results  on  this,  at  this  workshop. 

4.  Another  approach  being  followed  recognizes  the  importance  of 
full  mission  system  simulation  in  characterizing  workload 
scientifically.  FAA  and  NASA  are  working  together  on  the 
development  of  such  simulations  to  be  used  as  an  aid  in 
learning  more  about  establishment  of  objective  pilot  workload 
measures  t.o  augment  the  large  body  of  empirical  and  subjec¬ 
tive  information  which  now  exists.  Full  mission  system 
simulation  techniques  will  also  permit  improved  studies  of 
the  interface  between  the  pilot  and  the  ATC  system  where  many 
human  errors  originate. 
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SUMMARY 


During  the  next  decade  FAA  plans  to  conduct,  with  substantial 
support  f-rom  NASA,  DOD,  industry  and  the  universities,  a  compre¬ 
hensive  human  factors  program  which  will  address  the  causes  and 
impact  of  human  error  on  aircraft  and  ATC  system  operation,  and 
establish  criteria  and  guidelines  for  future  automated  system 
designs  which  will  minimize  the  occurrence  and  impact  of  human  error. 

The  program  will  reexamine  the  allocation  of  separation  assurance 
and  spacing  functions  between  the  cockpit  crew  and  the  controller, 
and  between  the  automated  systems  (both  in  the  aircraft  and  in  the 
ground  ATC  system)  and  the  human  operators  (controllers  and 
pilots.)  Since  the  human  operator  problems  and  human  involvement  in 
system  design  and  operation  are  the  major  thrust  of  this  activity, 
we  have  established  the  program  to  assure  that  all  elements  of  the 
aviation  community  have  a  voice  in  determining  our  program.  Perhaps 
most  important  to  the  activities  of  this  group  is  the  examination  of 
the  programs  underway  to  assess  the  impact,  of  workload  on  the  total 
system  operation  and  effectiveness.  I  invite  you  to  listen 
carefully  to  John  Fabry  the  day  after  tomorrow.  For,  I  believe,  his 
Techniques  for  Assessment  of  Pilot  Effectiveness  will  have  a  major 
impact  on  our  collective  ability  to  assess  the  impact  of  new  systems 
i ntroducti on . 
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Historical  Foundations  of  the  AFAMRL  WORKLOAD  PROGRAM 


Robert  D.  O'Donnell,  Colonel,  USAF 
Wright-Patterson  Air  Force  Base 


Historically,  the  concept  of  workload  has  been  a  morass  in  which  millions 
of  dollars  and  many  professional  reputations  have  been  lost.  This  has 
occurred  in  part  because  workload  is  a  deceptively  simple  concept  which 
people  easily  intuit,  but  cannot  easily  define.  Emanuel  Donchin 
has  described  workload  as  being  very  like  pornography.  Everyone  knows 
what  it  is  when  they  see  it,  but  it  is  extremely  difficult  to  get  a 
definition  which  is  generally  accepted.  In  fact,  one  can  find  at 
least  20  distinguishable  definitions  in  the  literature,  and  probably  100 
techniques  which  have  been  proposed  at  various  times  to  measure  workload. 

In  spite  of  these  problems,  the  term  workload  will  simply  not  go  away. 

Its  persistence  argues  persuasively  that  it  must  have  at  least  some 
potential  value  in  understanding  human  behavior.  The  reasons  for  this 
persistence  relate  to  the  massive  technological  changes  introduced  in 
the  last  several  years.  The  technoloy  being  used  in  today's  aircraft 
systems  has  introduced  such  complexity  into  the  aerospace  environment 
that  the  human  is  becoming  the  limiting  factor  in  systems  design. 

Whereas,  in  the  past,  physiological  limits  imposed  by  G  forces,  pressure, 
and  other  external  stresses  limited  the  aerosoace  vehicle,  it  is  becoming 
increasingly  frequent  to  find  that  the  human's  cognitive  limits  now 
restrict  potential  applications.  It  is  therefore  essential  that  ways  be 
developed  to  evaluate  the  load  that  existing  and  planned  systems  impose 
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on  the  operator.  Only  then  will  designers  be  in  a  position  to  avoid 
dangerous  overloads,  design  for  optimal  loading,  and  develop  new 
procedures  for  expanding  and  superseding  the  human  limits. 


Recognizing  this  need,  the  Workload  and  Ergonomics  Branch  of  the  Air 
Force  Aerospace  Medical  Research  Laboratory,  Wright-Patterson  AFB,  was 
formed  in  1979.  Its  mandate  was  to  establish  allowable  standards  of 
workload  in  all  Air  Force  systems,  as  well  as  to  develop  metrics  to  show 
compliance  with  such  standards.  As  part  of  this  effort,  this  Branch  has 
surveyed  the  history  of  workload  from  the  viewpoint  of  definitions  and 
techniques  and  a  general  conceptual  framework  has  been  evolved  for  the 
area  which  hopefully  can  provide  an  initial  foundation  and  structure  for 
the  definition  of  standards  and  the  development  of  metrics. 


"Definition"  of  Workload 

As  an  initial,  preliminary,  and  certainly  incomplete  "definition"  of 
workload,  the  following  statement  encompases  most  of  the  concepts  held 
leading  theorists  in  this  field.  Workload  is  an  hypothetical  construct 
which  conveniently  describes 'the  interactions  between  multiple  factors 
affecting  the  operator1, s  response  in  an  operational  system. 


by 


It  is  important  to  recognize  that  workload  is  an  hypothetical  construct 

which,  in  effect,  has  no  concrete  existence.  It  has  value  only  insofar 

as  it  conveniently  summarizes  many  diverse  things  under  one  concept. 

These  "diverse  things"  are  those  multiple  factors  which  affect  the 

operator's  response.  They  are  not,  however,  synonymous  with  the 

response.  Rather,  interactions  between  factors  produce  an  overall  effect 
on  the  operator  and  on  system  performance. 
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Obviously,  such  a  broad  and  incomplete  definition  has  value  only  if  the 
factors  underlying  them  can  be  identified,  and  if  metrics  to  assess  these 
factors  can  be  specified.  It  is  convenient  to  identify  three 
broad  categories  of  factors  which  contribute  to  workload,  and  then  to 
break  these  broad  categories  into  subcategories.  Hopefully,  measurement 
techniques  can  then  be  identified  which  will  map  onto  each  of  these  sub¬ 
categories  in  such  a  way  as  to  provide  an  overall  assessment  of  the' 
construct.  In  the  categories  to  follow,  the  major  classifications 
proposed  by  Johansen  are  utilized.  These,  in  themselves,  were  evolved 
from  several  other  theorists  (Morey,  1979).  They  consist  of:  (1)  task 
load,  (2)  operator  variables,  and  (3)  the  response.  In  the  sections  to 
follow,  each  of  these  will  be  considered  separately. 

Task  Load 

Task  load  is  defined  simply  as  all  of  the  behaviors  required  to  achieve 
the  task  goal.  It  is  extremely  important  to  begin  with  this 
aspect  of  the  workload  construct,  since  task  goal  and  the  behaviors 
required  to  achieve  it  provide  the  reason  for  existence  of  workload  in  the 
first  place.  It  can  be  seen  therefore  that  workload  analysis  must  start 
with  the  operational  goal  or  mission.  This  demands  that  there  must  be  a 
precise  definition  of  what  is  required  to  achieve  that  mission.  Without 
such  definition,  it  is  virtually  impossible  to  approach  questions  of  workload 
in  any  but  the  most  abstract  theoretical  sense. 

Specification  of  the  task  load  involves  at  least  three  major  subcategories. 
Perhaps  the  most  fundamental  of  these,  the  sensory  environment,  is  the 
one  which  is  most  frequently  overlooked.  The  temperature,  humidity, 
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vibration,  g-forces,  etc.,  in  which  a  behavior  occurs  significantly 
affect  the  difficulty  of  the  task.  Workload  measurement  procedures  have 
seldom  taken  such  factors  into  account,  and  no  established  methodology 
for  including  this  type  of  influence  in  the  overall  workload  equation  has 
been  developed. 

A  second  subfactor  of  the  mission  which  must  be  considered  is  the 
cognitive  demand  which  can  be  anticipated  in  the  task.  Cognitive  demand 
is  defined  as  those  internal  operations  necessary  to  achieve  the  task 
demands,  and  include  things  such  as  memory,  decisions,  and  other  mental 
operations  which  are  obviously  required  of  the  operator.  As  aerospace 
systems  evolve,  there  is  a  clear  trend  toward  an  increase  in  the  cognitive 
demands  of  tasks  at  the  expense  of  manual  control  tasks.  Yet,  few 
mission  description  techniques  are  capable  of  handling  such  cognitive 
demands,  or  even  clearly  specifying  which  cognitive  operations  are  required 
to  achieve  a  given  goal. 

The  final  and  most  obvious  task  load  subfactor  involves  motor  behaviors. 
These  consist  of  those  things  which  must  be  done  to  the  outside  environ¬ 
ment  in  order  to  achieve  the  task  goals  (such  as  control  behaviors, 
switch  activations,  etc.)  and  system  dynamics  which  impose  requirements 
on  the  operator  to  carryout  specific  error-nulling  responses.  Perhaps 
the  greatest  amount,  of  work  with  respect  to  aircraft  systems  and  workload 
has  been  done  in  one  aspect  of  motor  behavior,  namely,  manual  control. 
Elegant  models  can  permit  rather  precise  definition  of  the  control 
factors  critical  to  a  closed-loop,  continous  system.  Less  well  specified, 


but  still  extensively  studied,  have  been  the  discrete  performance  tasks 
required  from  the  operator.  Data  bases  consisting  of  task  completion 
times  and,  in  some  cases,  variability  estimates  are  available.  These  can 
be  incorporated  into  overall  mission  task  analyses. 


In  many  cases,  task  loading,  especially  the  motor  subfactor,  can  be  assessed 
by  task  analyses,  time  lines,  and  by  various  systems  models  which  have  been 
used  over  the  years.  These  analyses  and  models  essentially  ask  whether  the 
job  can  be  done.  More  simply,  they  ask  if  there  is  enough  time  to  do  all 
that  has  to  be  done.  As  such,  task  analyses  are  critical  to  any  workload 
analysis  and  constitute  the  starting  point  in  deciding  whether  the  system  is 
even  feasible. 

However,  in  many  cases,  task  analyses  and  systems  models  have  been  viewed 
as  the  final  workload  measure,  and  have  been  presented  as  the  definitive 
evidence  that  a  system  does  or  does  not  meet  workload  standards.  Aircraft 
manufacturers  have  relied  heavily  on  these  procedures  to  demonstrate  to  the 
FAA  that  a  proposed  system  is  certifiable  with  respect  to  workload.  Even 
ignoring  the  fact  that  such  models  have  obvious  weakness  in  the  area  of 
sensory  and  cognitive  factors.  Task  analyses  almost  always  fail  to  consider 
the  operator  as  a  major  factor  in  system  operation.  Such  failure  can  result 
in  a  system  which,  while  flyable  under  optimal  human  conditions,  is  not 
designed  for  the  real  operational  enviornment.  In  that  environment,  the 
operator  is  a  major  determinant  of  the  task  demands  as  well  as  the  major 
mechanism  on  which  those  demands  act. 
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To  present  this  more  forcefully.  Figure  1,  shows  that  there  is  a  two-way 
interaction  between  the  task  demands  and  the  operator.  In  effect,  the  task 
demands  and  the  operator  together  form  the  person-machine  system.  Any 
workload  metrics  must  consider  this  system  as  a  whole.  No  existing  model 
adequately  achieves  this  goal  and,  at  best,  such  models  or  task  analyses 
statistically  approximate  the  mutual  interaction  between  task  demands 
and  operator.  Since  little  is  known  about  whether  such  interactions  are 
additive,  multiplicative,  or  much  more  complex,  such  approximations  are 
always  crude. 
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FIGURE  1 


Operator  Factors 


If  the  operator  must  be  considered  as  part  of  the  overall  system,  those 
subfactors  constituting  this  major  influence  must  be  identified.  Clearly, 
the  operator's  physical  capacity  is  a  basic  subfactor  in  this  category. 
Although  in  most  aircraft  environments  adequate  selection  makes  this  a 
less  crucial  variable,  it  assumes  considerably  greater  importance  if  the 
operator  may  be  decremented  due  to  injury  or  toxic  exposure.  In  such 
cases,  workload  analysis  must  consider  the  interactions  generated  by  a 
less  than  optimal  physical  condition.  Analogously,  mental  capacity  must 
be  evaluated.  Again,  while  selection  in  most  aircraft  enviornments 


assures  adequate  initial  mental  capacity,  long-term  changes  due  to  stress, 
fatigue  or  other  decrementing  conditions  must  be  anticipated.  Similarly, 
the  subfactor  of  emotional  condition  must  be  specified  with  respect  to 
the  workload  environment.  These  three  categories  of  physical,  mental, 
and  emotional  state  have  not  usually  been  considered  necessary  in 
workload  assessment.  Yet,  in  many  cases  they  constitute  the  questions 
of  greatest  interest  to  the  operators  themselves,  who  are  aware  of 
transient  changes  in  these  factors  and  wish  to  be  reassured  that  the 


workload  can  be  handled  under  decremented  conditions. 


The  most  obvious  operator  subfactor  to  be  evaluated  is  the  effect  of  task 
demands  on  the  operator's  mental  resources.  Given  adequate  physical,  mental, 
and  emotional  capacity,  there  is  still  the  question  of  the  short  and  long  term 
effects  of  performance . in  the  enviornment  on  the  operator's  reserve  mental 
capacity,  on  the  strategies  employed  in  meeting  the  task  demands,  and  on 
questions  of  attention  and  resource  allocation  over  time.  Simply  stated, 


an  aircraft's  tasks  may  be  do-able  by  the  rested,  trained,  motivated  pilot, 
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but  they  may  not  leave  any  margin  for  safety,  and  sustained  performance 
over  a  period  of  time  may  significantly  affect  what  margin  existed 
originally.  For  such  questions  of  mental  resources,  whole  series  of 
procedures  have  been  developed.  Briefly,  these  procedures  can  be  viewed 
as  falling  into  three  major  classes. 

Subjective  measures  are  those  in  which  the  operator  is  simply  asked  to 
evaluate  either  his  or  her  own  workload  or  that  of  another  operator. 

These  measures  are  based  on  the  assumption  that  the  human  being  is  the 
best  observer  of  all  the  interactions  between  factors  entering  into  the  - 
workload  construct.  Many  types  of  subjective  measures  have  been  used, 
including  modifications  of  the  familiar  Cooper-Harper  scale,  as  well  as 
newer,  advanced  techniques  such  as  conjoint  analysis. 

Many  believe  that  subjective  assessment  techniques  are  the  best  single 
measure  of  workload.  Certainly  they  have  good  face  validity  and  have 
produced  the  best  results  in  the  past.  One  must  be  aware,  however,  of 
their  limits.  Subjective  techniques  may  not  be  very  helpful  in  some 
situations,  particularly  where  the  purpose  is  to  diagnose  the  cause  of 
the  workload  problem.  Although  pilots  may  frequently  be  able  to  elaborate 
on  a  subjective  rating  and  to  provide  useful  insights  into  the  source  of 
the  workload  problem,  it  is  not  always  desirable  to  place  heavy  reliance 
on  such  analyses.  Individual  motivations,  capacities,  or  preference  may 
bias  results  and  can  lead  to  false  conclusions  (for  instance,  the  "old 
boy  syndrome"  may  perpetuate  poor  but  familiar  procedures).  In  spite  of 
this  limitation,  however,  as  Sheridan  and  Jex  have  maintained,  subjective 
measures* will  be  extremely  important  for  a  long  time  to  come,  and  consider¬ 
able  efforts  should  be  expended  in  improving  these  techniques. 
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Behavioral  measures  consist  of  two  major  types.  Primary  task  measures 


assess  the  person's  performance  within  the  system.  More  will  be  said 
about  this  type  of  measurement  later.  Secondary  task  measures  involve 
the  imposition  of  an  additional  task  on  the  operator  in  order  to  "load" 
the  person  to  maximum  capacity.  The  break  point  in  primary  task  perfor¬ 
mance  can  be  used  as  a  measure  of  workload,  or  the  secondary  task  can 
be  used  to  provide  an  independent  assessment  in  its  own  right.  Many  types 
of  secondary  tasks  are  available.  The  workload  assessment  device  (WAD) 
currently  used  at  Patuxent  River,  Maryland  by  the  Navy  was  initially 
developed  at  AFAMRL,  and  represents  a  good  example  of  this  type  of 
technique  currently  being  used  in  operational  environments  with  great 
success. 

At  AFAMRL,  primary  emphasis  has  been  on  the  development  of  non-intrusive 
secondary  tasks.  In  particular,  one  type  of  task  has  generated  consider¬ 
able  interest.  This  has  been  called  the  embedded  secondary  task.  In 
this  procedure,  a  part  of  the  primary  task  (such  as  the  communications 
subtask  in  a  flying  task)  is  taken  into  the  laboratory  and  quantified 
with  respect  to  workload.  Once  the  workload  involved  in  this  subtask 
is  well  defined,  it  can  then  be  reintroduced  into  the  primary  task.  As 
far  as  the  operator  is  concerned,  no  artificial  task  has  been  introduced, 
and  yet  a  quantified  secondary  task  is  being  presented,  thus  yielding  the 
advantages  of  the  secondary  task  methodology.  This  procedure  has  been 
developed  for  the  A-10  aircraft  communications  task,  and  is  currently 
being  tested  in  a  simulator  environment.  Current  plans  call  for  it 
to  be  available  in  the  near  future  for  field  use. 

The  final  technique  for  assessing  operator  variables  consist  of  neuro¬ 
psychological  procedures  such  as  the  cortical  evoked  response,  heart  rate 
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measures,  muscle  measurements,  blink  rate  and  eye  movements,  etc.  which 
can  measure  the  effect  of  task  load  directly  on  the  person.  It  is 
important  to  recognize  that  these  procedures  are  not  measuring  physiology 
in  any  real  sense,  nor  are  they  looking  at  the  medical  condition  of  the 
pilot.  Rather,  they  look  at  the  brain,  muscle,  and  heart  responses  which 
are  given  by  the  operator.  In  this  sense,  they  are  no  different  than  a 
verbal  report  or  a  motor  response.  However,  they  are  obtained  more 
quickly  and,  hopefully,  more  reliably.  We  are  currently  testing  a  neuro¬ 
psychological  test  battery  which  contains  eleven  such  standardized 
procedures.  These  are  being  validated  in  Air  Force  environments,  and 
we  hope  to  have  at  least  some  of  them  available  for  flight  test  in  the 
near  future. 

With  these  subjective,  behavioral,  and  neuropsychological  procedures,  we 
hope  to  assess  the  operator's  interaction  with  the  task  load  more 
precisely,  and  to  provide  the  data  base  necessary  to  permit  adequate 
modeling  of  the  person-machine  interaction. 

Response  Factors 

Even  when  the  above  goal  has  been  achieved,  there  is  still  another  major 
factor  influencing  the  overall  workload  equation  which  must  be  considered 
The  response  is  really  the  major  concern  in  Air  Force  systems.  Certainly, 
one  would  expect  that  the  response  will  be  correlated  with  the  workload 
imposed  on  a  person-machine  system.  In  fact,  many  would  propose  that 
system  performance  (primary  task  performance)  be  made  the  ultimate  workload 


63 


measure.  This  argument  says  simply  that  if  there  is  no  decrement  in 
system  performance,  workload  is  acceptable.  The  problem  with  this  logic 
is  that  the  response  is  not  always  manifested  in  short-term  system 
performance.  In  many  situations,  the  person  performs  perfectly  until 
catastrophic  failure.  For  this  reason,  it  is  more  precise  to  subdivide 
the  response  into  two  elements.  System  performance,  as  defined  above,  is 
one  of  those  elements.  The  other  is  the  "cost"  to  the  operator.  This  is 
defined  as  the  decrement  to  an  operator's  capacity  as  a  function  of 
maintaining  performance.  While  the  workload  of  the  person-machine 
system  determines  the  response,  this  response  can  be  manifested  either  in 
a  performance  effect  or  in  an  effect  on  an  operator's  capability.  This, 
in  turn  feeds  back  into  the  system,  as  shown  in  Figure  2.  If  performance 
is  affected,  it  affects  the  tasks  to  be  done  in  order  to  achieve  the 
mission  goal.  On  the  other  hand,  if  the  person's  capacity  changes  it 
affects  the  operator's  factors.  In  either  case  one  can  conceptualize 
essentially  all  of  the  various  interactions  constituting  the  workload 
construct  within  this  dynamic  framework. 


Implications  for  Workload  Assessment 

Using  the  above  framework,  several  principles  of  workload  assessment 
appear  evident.  First  of  all,  it  should  be  clear  that  a  measure  of  any 
one  of  the  factors  shown  in  Figure  2  is  not  in  itself  a  workload  measure, 
unless  all  other  factors  and  interactions  are  specified  or  controlled. 

In  other  words,  no  single  factor  is  the  workload .  The  confusing  and 
frequently  contradictory  results  found  in  the  workload  literature  have 
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WORKLOAD  FRAMEWORK 
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FIGURE  2 


most  often  ben  due  to  the  fact  that  individuals  have  measured  one  factor 
while  failing  to  control  or  account  for  all  the  others.  The  practical 
impact  of  this  principle  is  that  a  workload  program  must  be  multi¬ 
dimensional,  and  must  include  techniques  for  approaching  all  of  the 
factors  involved.  In  addition,  any  attempt  to  measure  workload  in  a 
system  must  give  considerable  attention  to  discussing  potential  inter¬ 
actions  between  factors  which  are  not  measured  in  that  particular  study. 

It  is  possible,  for  example,  to  do  a  perfectly  adequate  workload  assess¬ 
ment  without  considering  decremented  environmental  conditions.  However, 
in  such  a  case,  it  is  necessary  to  clearly  define  the  limits  of  environ¬ 
mental  conditions  to  which  any  results  would  be  applicable.  Awareness  of 
such  factors  would  preclude  a  great  deal  of  overinterpretation  of  workload 
studies,  while  still  permitting  valuable  data  to  be  collected  upon  which 
decisions  can  be  made. 

A  second  implication  of  the  above  framework  is  that  the  search  for  a 
"holy  grail"  of  workload  assessment  should  probably  be  abandoned.  No 
single  measure,  including  subjective  assessment,  will  ever  be  able  to 
adequately  assess  all  of  the  multiple  factors  and  interactions.  To  this 
end,  the  AFAMRL  Workload  Program  has  dedicated  itself  to  development  of 
a  broad  range  of  metrics  which  can  be  included  in  test  batteries.  In 
this  way,  the  historical  lessons  learned  from  past  attempts  to  assess 
workload  can  be  applied  on  a  case-by-case  bases  to  specific  workload 
questions.  Tests  can  be  chosen  from  the  batteries  which  answer  the 
question  being  asked,  rather  than  attempting  to  answer  amorphous  general 
workload  question*,  or  answering  a  question  which  was  not  asked  in  the 
first  place,  simply  because  a  measure  is  available. 
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In  conclusion,  nothing  that  has  been  discovered  about  workload  in  the 
last  ten  years  has  changed  its  basically  complex  nature,  nor  has  it 
revealed  a  simplistic  way  of  measuring  it.  However,  significant 
progress  has  been  made  in  defining  what  the  various  critical  elments  of 
workload  are,  and  in  pointing  out  those  elements  which  can  be  measured 
at  the  present  time,  as  well  as  those  which  will  require  new  metrics 
for  their  measurement.  At  AFAMRL,  such  metrics  are  being  developed, 
and  it  is  hoped  that  they  will  be  applicable  to  all  environments. 

There  is  considerable  optimism  that  such  a  goal  is  achievable,  and  we 


intend  to  achieve  it. 


Pilot  Models  for  Flight  Control  Analysis 
R.  0.  Anderson 
Flight  Dynamics  Laboratory 


Summary :  After  a  brief  definition  of  "pilot  models",  the  two  most 

common  models  ("classical"  and  "optimal")  are  briefly  described. 

The  basic  limitations  and  advantages  of  each  are  summarized.  Next, 
some  of  the  potential  uses  of  these  models  are  discussed  along  with 
a  few  non-aircraft  oriented  applications.  The  available  data  base 
to  confirm  models  is  reviewed  and  it  is  concluded  that  despite  many 
very  successful  applications  to  date,  overall  understanding,  use, 
and  actual  in-flight  data  confirmations  are  limited. 

Pilot  Models:  As  used  here,  a  "pilot  model"  is  some  form  of  mathe- 
matical  expression  that  attempts  to  portray  how  the  human  operator 
behaves  in  certain  "control"  situations.  Perhaps  the  first  crude 
model  was  in  N.  Minorsky's  mind  when  he  wrote,  in  1922  (Ref.  1) 

"It  has  often  been  stated  that  the  human  intuition  of  the 

helmsman  cannot  be  replaced  by  any  mechanical  contrivance 

whatever  its  mature  may  be.  Such  a  standpoint  seems  to  be 

erroneous . " 

He  then  proceeds  to  describe  the  human  operator  (helmsman)  action 
as  responding  to  ship  yaw  plus  yaw  rate.  In  short,  a  verbal  de¬ 
scription  of  the  helmsman  as  a  linear  control  system  that  we  could 
represent  with  a  differential  equation,  or  "pilot  model". 

No  attempt  will  be  made  here  to  trace  the  detailed  history  of 
mathematical  models  of  human  behavior;  instead,  the  reader  should 
consult  Refs.  2  and  19  for  such  a  discussion.  The  purpose  of  this 
brief  review  is  to  emphasize  a  few  milestones  in  the  story  and,  if 
possible,  bring  the  current  status  into  perspective,  at  least  from 
the  author's  viewpoint. 

Classical  Servo  Model;  Dubbed  "classical"  this  model  was  the  eai.lie&t 
to  gain  widespread  credibility.  It  simply  assumes  the  pilot's  input- 
output  response  for  certain  tasks  is  similar  to  a  "good  servo"  system. 
This  reversal  of  purpose  from  Minorsky  (who  wanted  to  build  a  "servo" 
that  operated  like  his  "model"  of  the  human)  was  made  possible  by 
some  35  years  of  linear  control  theory  development.  Now,  the 
classical  model  is  simply  a  fundamental  application  of  undergraduate 
linear  control  theory. 

The  tasks  mentioned  above  must,  however,  reduce  the  human 
operator  to  somewhat  of  a  "monkey"  status.  That  is,  the  parameters 
to  be  controlled  must  be  well  known  (e.g.,  hold  aircraft  pitch 
attitude),  the  inputs  or  disturbances  must  be  "random  appearing", 
the  operator  must  devote  full  attention  to  the  task,  and  the  pilot/ 
aircraft  system  must  represent  an  equivalent  single-input  and  single¬ 
output  linear  compensatory  system.  Work  has  been  done  to  relax  every 
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one  of  these  restrictions  with  some  varying  degrees  of  success,  but 
for  the  most  part,  they  must  all  hold  for  a  "confident"  application. 

The  good  news  for  modelers  is  the  above  "monkey"  restriction 
doesn't  seem  to  preclude  application  to  the  task  of  flying  a  real 
aircraft.  This  is  also  the  bad  news  for  pilots. 

Figure  1  shows  the  heart  of  the  classical  model.  With  the 
pilot  quasi-linear  describing  function  given  as  Yp,  and  the  aircraft 
output  (m)  to  input  (c)  transfer  function  fiven  as  Yc,  the  name  of 
the  game  is  for  the  pilot  to  vary  his  response  to  the  error  signal 
(e)  such  that  the  product  YpYc  becomes  that  of  a  pure  integrator  (k/s) 
in  the  region  of  crossover.  The  latter  is  the  frequency  range  where 
the  YpYc  amplitude  ratio  becomes  unity  on  a  Bode  plot.  This  "goal 
of  adaption"  is  one  version  of  what  it  takes  to  make  a  "good  servo" 
(Ref.  2).  Does  it  work?  Yes  indeed.  In  fact,  the  author  has  seen 
measurements  that  indicate  this  not  only  in  the  case  for  good  old 
U.  S.  of  A.  pilots,  but  "trackers"  from  several  other  countries.  If 
you  still  don't  believe,  check  Ref.  3  which  probably  still  represents 
the  most  carefully  conducted  and  complete  experimental  work  in  the 
pilot  model  parameter  identification  area. 

As  an  interesting  aside,  the  experiments  in  Ref.  3  utilized  a 
data  reduction  scheme  using  common  utility  company  watt-hour  meters 
because  they  were  the  most  cost-accuracy  effective  analog  computers 
at  that  time .  ~ 

Despite  numerous  applications  that  many  consider  an  unqualified 
success,  the  classical  model  still  has  its  limitations.  In  short, 
only  single-input,  single-output  tasks  are  fully  understood  and 
validated.  The  task  must  be  well  known  and  clearly  defined,  and  must 
satisfy  the  "monkey"  conditions.  Nonetheless,  the  classical  models 
have  been  applied  to  some  extremely  complex  tasks. 

On  the  other  h&nd,  the  biggest  advantage  of  the  model  are  its 
simplicity  and  the  fact  that  it  works! 

Optimal  Pilot  Model:  Before  proceeding  with  this  discussion  a  word 
of  caution  is  in  order.  While  assisting  another  engineer  at 
Wright-Patterson  Air  Force  Base  with  a  problem  of  evaluating  turbu¬ 
lence  effects  on  a  new  proposed  aircraft  payload,  it  was  suggested 
that  a  "pilot  model"  be  used  to  analytically  control  the  unstable 
spiral  mode  of  the  aircraft.  The  engineer's  reaction  was  that  this 
could  not  be  done  without  "picking  one  company's  or  the  other's 
model",  and  this  was  a  no-no.  Unfortunatley ,  pilot  models  may  have 
become  associated  with  one  person  or  company.  Therefore,  keep  in 
mind  the  type  we  are  discussing  all  assume  the  human  operator  acts 
like  a  "good  servo".  How  you  design  the  good  servo  is  the  key 
"individual"  distinction  between  the  model  (see  Ref.  19  for  a  com¬ 
parison  of  the  classical  and  optimal  models) . 

Now,  on  to  the  optimal  model.  Few  know  that  this  now  famous 
creature  was  born  in  the  lounge  of  the  old  Boston  Hilton.  Seems  BBN 
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(Ref.  4)  was  having  trouble  using  the  "classical"  model  under  an 
Air  Force  contract  to  apply  pilot  models  to  aircraft  display  require¬ 
ment  determination.  Viewing  the  pilot  as  an  optimal  controller  had 
been  suggested  by  the  author  (and  other)  in  an  IEEE  Journal  Note. 
Jerry  Elkind,  then  of  BBN,  suggested  this  be  attempted  under  the 
contract  one  night  at  an  informal  "conference".  The  answer  was  yes, 
but  good  luck  selecting  the  performance  index.  That  started  it  all. 

The  optimal  control  model,  shown  in  Fig.  2  simply  assumes  the 
pilot  has  some  mental  "performance  index",  and  being  the  "good  servo" 
that  he  is,  he  minimizes  the  index.  This  index  is  a  quadratic  fun¬ 
ction  of  aircraft  state  (x)  and  control  input  (u) .  Once  the  index 
is  available,  a  common  control  theory  "meat  grinder"  will  produce 
the  optimal  controller  which  becomes  the  assumed  pilot  model.  The 
net  result  is  the  same  as  the  classical  approach;  namely,  "prediction 
of  closed-loop  performance  and  pilot  model  parameters".  Foes  argue 
that  no  "real"  pilot  knows  a  "quadratic  performance  index"  from  a 
flame-out,  that  the  results  with  a  single  performance  index  will 
match  observations,  etc.  Friends  say  it's  not  a  "real"  model,  but 
does  predict  performance  and  it  handles  multi-input,  multi-output 
problems  readily  (the  above  meat-grinder  is  very  powerful  in  this 
regard).-/  Suffice  -it  to  say,  it's  another  way  to  design  a  "servo" 
(pilot  model),  and  it  also  works. 

Other  Models:  Again,  no  attempt  will  be  made  to  list  the  dozens  of 
derivatives  of  the  above  two  models  that  have  been  postulated  over 
the  years.  Reference  2  contains  references  to  many.  Typical  of 
these  is  the  "dual-loop"  model  in  Refs,  5  and  6,  and  the  "step¬ 
tracking"  model  in  Ref.  7.  The  interesting  thing  about  both  of  these 
is  that  they  are  being  used  in  an  attempt  to  predict,  or  evaluate, 
flying  qualities.  This  was,  in  fact,  the  noble  goal  of 
Charlie  Westbrook  in  the  AF  Flight  Control  Laboratory  back  in  1954 
when  he  sponsored  much  of  the  early  work  in  pilot  modeling  that  he 
hoped  would  see  fulfillment  in  5  or  10  years. 

The  details  of  model  development,  applications,  and  general 
engineering  acceptance  are  too  numerous  to  cover  here.  Reference  2 
and  the  "Annual  Manual"  proceedings  (e.g..  Ref.  8)  cover  much  of  the 
story,  both  successes  and  setbacks  (e.g.,  see  Refs.  40,  44  -  54  in 
Ref.  2) .  The  Annual  Manual  really  started  when  the  Air  Force 
decided  to  have  the  few  researchers  in  the  area  (sponsored  by  the 
AF  and  Navy)  meet  once  a  year  to  discuss  progress.  In  fact,  one 
meeting  was  adjourned  to  the  author's  house  for  beer,  pizza,  and 
further  discussions.  Now  the  annual  meetings  attract  75  to  100 
people . 

The  cast  in  these  developments  is  also  too  large  for  much 
individual  acknowledgement.  Therefore,  at  the  risk  of  raising  the 
wrath  of  those  I  leave  out,  the  classical  U.  S.  leaders  that  must  be 
mentioned,  and  are  still  in  the  business,  are:  McRuer,  Krendel,  and 
Graham.  Those  moving  on  to  other  areas  include:  Tustin,  Elkind, 
Russell,  Sobczyk,  Phillips,  and  Weiss. 
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The  U.  S.  optimizers  have  held  in  a  bit  better  with  those 
still  making  a  sound  living  on  the  "optimal  servo":  Baron,  Levison, 
Kleinman,  Hess,  and  most  recently  Dave  Schmidt  of  Purdue.  Fallen 
by  the  wayside  are  Elkind,  Peter  Falb  (who  extended  modern  control 
theory  which  at  the  time  could  not  include  the  known  pilot  pure 
time  delay),  and  Jim  Dillow  (Ref.  16)  my  partner  in  crime  who  made 
pilot  modeling  "fun". 

Pilot  Model  Uses:  It  would  be  nice  to  say  at  this  point  that  every 
high  school  graduate  understands,  and  uses,  pilot  models.  This  just 
isn't  so.  While  a  large  number  of  applicantions  do  exist  (again  see 
Ref.  2)  they  have  been  accomplished  in  the  U.  S.  by  only  a  few 
people.  Fortunately  in  Europe  (where  a  European  Annual  Manual  was 
recently  started)  the  situation  seems  to  be  different.  In  any 
event,  a  few  applications  to  aircraft  control  are  covered  here. 

Perhaps  the  most  worked,  and  reworked,  area  of  application  is 
"human  gun  tracking".  In  fact,  the  early  work  of  Tustin  in  1947 
(see  Ref.  2)  concerned  this  task.  About  the  same  time  Phillips  and 
Sobczyk  also  analyzed  this  task  (Ref.  9)  and  even  adjusted  system 
parameters  to  minimize  mean-squared-error;  which  Schmidt  is  doing 
today  for  the  air-to-air  gun  tracking  task  (Ref.  10). 

The  next  application  area,  prediction  of  flying  qualities, 
motivated  much  of  the  early  work  in  pilot  modeling.  However,  the 
number  of  serious  attempts  to  quantitatively  predict  numerical  values 
of  pilot  acceptance  (e.g.,  Cooper-Harper  rating  scale)  are  very  few. 

In  most  cases,  the  basic  concept  is  that  pilot  acceptance  is  degraded 
as  closed- loop  performance  degrades  and/or  "workload"  increases. 

The  key  issue  is  what  constitutes  a  suitable  measure  of  "performance" 
and  "workload"? 

Reference  11  represents  my  first  contribution  to  the  rating 
prediction  confusion.  Basically,  the  conjecture  is  that  "performance" 
is  some  measure  of  closed- loop  root-mean-square-error  values,  and 
"workload"  is  generated  lead  (value  of  lead  time  constant  in  the 
pilot  model) .  The  kicker  is  the  pilot  model  parameters  are  adjusted, 
within  measured  limits,  to  minimize  pilot  rating  (best  overall 
acceptance) . 

The  concept,  called  "Paper  Pilot",  is  actually  vwey  simple, 
although  the  original  digital  computer  program  to  implement  this 
approach  was  fairly  large  and  complex.  This  fact,  along  with  an 
apparent  virtue  of  the  approach  that  backfired  (i.e.,  completely 
automated  flying  qualities  numerical  prediction)  may  have  hindered 
acceptance.  The  latter  point  may  follow  the  story  of  the  early  non- 
acceptance  of  pre-mixed  cake  mixes,  which  prevailed  until  the  con¬ 
cept  was  changed  and  the  housewife  was  required  to  add  an  egg  to  the 
mix.  With  this  "personal"  touch,  all  went  well. 

Shortly  after  the  Paper  Pilot,  Neal  and  Smith  at  Calspan 
(Ref.  12)  proposed  longitudinal  "criteria"  for  flying  qualities 
that  also  used  a  pilot  model.  Again,  ratings  were  correlated  with 
closed-loop  performance  (in  this  case  it  was  damping  ratio  which 
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is  related  to  the  root-mean-square  values  in  the  Paper  Pilot)  and 
pilot  "workload"  (i.e.  ,  pilot  model  lead,  or  lag,  time  constant 
with  the  former  the  same  as  Paper  Pilot) .  The  model  parameters  are 
"egged"  into  line  by  trial-and-error  to  achieve  a  closed-loop  band¬ 
width  with  a  limit  on  low  frequency  "droop"  in  the  aircraft  pitdh 
transfer  function  Bode  plot.  The  results  are  plotted  against  con¬ 
stant  pilot  rating  contours  to  evaluate  the  configuration  in 
question. 

Although  the  Neal-Smith  criterion  appears  to  be  receiving 
greater  acceptance  than  the  automated  Paper  Pilot  (the  former 
apparently  has  the  "egg") ,  neither  have  made  it  to  Spec-dom  (Ref.  13) 
and  use  of  pilot  models  for  rating  prediction  remains  an  amature 
sport. 


While  the  flying  qualities  community  continues  to  rush  into 
the  twentieth  century,  the  grand-daddy  of  uses,  gun  tracking  per¬ 
formance  improvement,  may  be  coming  into  its  own.  Schmidt  (Ref.  10) 
has  applied  the  optimal  model  to  the  problem  quite  successfully,  and 
has  generated  a  great  deal  of  renewed  interest  in  this  subject. 

But,  what  about  other  applications.  Well,  aircraft  deplay 
selection  and  location  seems,  or  seemed,  a  natural  and  the  optimal 
model  came  about  from  such  an  application  (Ref.  4) .  Much  more  could 
be  done  in  this  area. 

In  a  similar  manner,  controller  design  might  be  another  appli¬ 
cation  for  extended  models  (Chapter  IV  of  Ref.  2) .  At  least  one 
such  case  is  known.  A  collegue  in  the  Army  called  me  about  yet 
another  gun  tracking  situation  in  a  tank.  The  "pilot  model"  produced 
markedly  better  performance  than  real  gunners.  After  checking  their 
model  and  finding  no  obvious  boo-boo's,  I  asked  about  the  controller. 
It  was  a  two-axis  "coolie  hat"  on  top  of  a  vertical  post  operated  by 
the  gunner's  thumb.  Try  it,  it's  awful!  I  told  him  to  replace  this 
with  a  good  force  stick  and  try  again.  He  called  a  few  weeks  later 
and  said  now  all  was  well.  Wonder  if  they  ever  used  the  new  con¬ 
troller  in  a  real  tank  based  on  a  phone  call? 

Another  long  overdue  application  is  simulation  validation. 
Closely  related  to  the  flying  qualities  use,  if  the  pilot  models 
can't  fly  it,  the  simulation  is  in  trouble.  I  was  asked  to  do  this 
once  in  our  large  hybrid  simulation  facility.  It  was  kind  of  weird 
using  a  half-dozen  operational  amplifiers  to  validate  a  several 
hundred  op.  amp/32K  digital  simulation! 

A  related  incident  dealt  with  in-flight  simulation.  Back  in 
1970  the  Navy  was  very  proud  of  an  F-8D  being  operated  as  a  variable- 
stability  aircraft.  I  took  the  oldest  set  of  flying  qualities  data 
for  the  longitudinal  axis  I  could  find  that  had  pilot  model  "data" 
(1958),  and  created  the  Pitch  Paper  Pilot  (Ref.  14).  This  model  was 
then  used  to  predict  (validate)  the  latest  F-8D  results  (Ref.  15). 

Some  of  the  correlations  were  good,  some  bad.  However,  a  pilot  model 
reaction  delay  time  of  0.42  seconds  had  to  be  used  in  the  model  to 
get  even  reasonable  results,  vs.  a  value  of  about  0.2  usually  used 
when  motion  cues  are  present.  About  eight  months  later  I  learned  the 
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Navy  had  "found"  a  pre-filter  in  the  F-8D  (I  don't  know  who  "lost" 
it).  In  turn,  it  had  a  2.5  rad/sec  break  point  which  produced  a 
phase  lag  that  almost  exactly  accounted  for  the  required  0.22 
second  increase  in  pilot  model  delay.  Paper  Pilot  had  "invalidated" 
an  in-flight  simulation  with  a  model  based  on  data  from  12  years 
earlier! 

Finally,  we  may  someday  reach  Minorsky's  goal  and  model  the 
human  on  our  way  to  forming  a  "true"  adaptive  flight  control  system. 
McRuer  has  often  called  the  pilot  the  "vocal  adaptive  controller". 
This  use  is  self-evident. 

Many  other  non-aircraft  applications  are  available,  ranging 
from  auto  driver  models  to  Minorsky's  problem,  the  helmsman.  The 
Netherlands  has  tried  the  latter  (unsuccessfully)  as  well  as  bike 
rider  (successfully)  applications.  Shopping  cart  driven  models 
anyone? 

Pilot  Model  Parameter  Data;  Sooner  or  later  this  long  "short  story" 
has  to  cover  flight  test  if  it  is  to  be  presented  at  a  meeting  at 
AFFTC.  Now  is  the  time. 

Ground-based  simulation  data  to  support  both  the  classical  and 
optimal  pilot  models  are  quite  readily  available  for  single-loop 
tasks  with  simple  controlled  elements  (e.g..  Ref.  3).  Many  of  the 
optimal  cases  include  only  closed-loop  performance  correlations,  but 
a  few  convert  the  complex  pilot  model  into  Bode  plot  form  for  com¬ 
parison  with  measured  describing  functions  or  other  parameter  data. 

When  it  comes  to  "real"  aircraft  dynamics,  the  data  bank  dries 
up.  However,  we  did  conduct  some  pilot  describing  function  experi¬ 
ments  using  the  B-l  simulation  at  the  Flight  Dynamics  Laboratory. 

The  really  unique  aspect  of  the  effort,  however,  was  the  fact  we  had 
help  from  a  Navy  Post  Graduate  School  Professor  and  used  an  Army 
describing  function  analyzer! 

As  for  pilot  model  parameters  measured  in-flight,  they  are  few 
and  far  between.  After  some  earlier  initial  attempts  (e.g. ,  Ref.  20) 
Newell  (Ref.  17)  obtained  ground  and  in-flight  simulation  results  for 
roll  tracking.  This  feat  was  repeated  in-flight  somewhat  later  with 
the  then  new  NASA  Jet-Star.  Shortly  afterward,  Wingrove  (Ref.  18) 
reduced  some  actual  data  from  a  Gemini  retro-rocket  firing  under 
manual  attitude  control.  This  one  experiment  almost  put  our  space 
age  data  bank  (1957-present)  even  with  our  aeronautics  (1903-present) 
bank.  Not  very  impressive,  especially  in  light  of  modern  instrumen¬ 
tation/data  reduction  improvements. 

The  bottom  line  of  this  section  is  obvious.  The  workshop  for 
which  this  was  written  (AIAA  Workshop,  "Flight  Testing  to  Identify 
Pilot  Workload  and  Pilot  Dynamics")  is  clearly  long  over-due  in 
respect  to  pilot  dynamics  in  general,,  and  pilot  parameter  identifi¬ 
cation  in  particular 
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Conclusions :  Despite  what  many  (including  the  author)  feel  have 

been  unqualified  successful  devlopments  and  applications  of  pilot 
models,  only  limited  applications  and  understanding  are  really 
evident  in  the  U.  S-  This  was  driven  home  to  me  at  a  recent  AGARD 
Flight  Mechanics  Panel  Symposium  where  one  attendee,  at  a  break, 
suggested  that  very  few  in  the  audience  understoo'd  the  last  paper 
which  was  based  on  pilot  models-  When  I  asked  him  what  we  could  do 
about  the  situation,  he  had  no  concrete  advise.  Perhaps  this  work¬ 
shop,  and  this  paper,  will  kindle  some  new  interest  and  new  practi¬ 
tioners.  If  they  do,  the  field  is  still  wide  open.  We  need  multi¬ 
loop  and  in-flight  pilot  model  parameter  data,  extensions  of  the 
models  to  tracking  plus  discrete  decision  tasks,  and  more  example 
applications  with  actual  aircraft  comparisons  to  increase  general 
understanding  and  acceptance. 

As  for  the  future,  well  Minorsky  and  Tustin,  we're  still 
trying,  and  we  may  have  some  additional  help  pretty  soon. 
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INTRODUCTION 


This  paper  reviews  some  results  of  NLR  research  on  pilot  performance 
and  workload  both  in  manual  control  and  monitoring  tasks .  A  substantial 
part  of  the  research  has  been  devoted  to  the  description  of  human  control 
behavior  and  task  conditions  in  terms  of  modern  control  theory.  The  prin¬ 
cipal  results  of  this  modelling  approach  are  considered  in  the  first  part 
of  this  paper.  It  is  shown  that  the  modelling  approach  can  provide  an 
adequate  framework  for  analysis  of  pilot  dynamics  and  workload. 

The  second  approach,  discussed  in  the  subsequent  part  of  this  paper,  deals 
with  the  assessment  of  pilot  workload  by  means  of  physiological  parameters. 
The  results  of  an  in-flight  experiment  reflect  a  good  correlation  between 
measures  of  heart  rate  and  respiration  frequency,  model  results,  sub¬ 
jective  ratings,  overall  performance  and  control  activities. 


MODELLING  APPROACH 


General 

During  the  last  decades  considerable  research  effort  has  been 
devoted  to  the  development  of  mathematical  tools  for  the  analysis  of 
human  control  behavior.  Among  several  different  approaches  towards  human 
operator  modelling  the  optimal  control  model  has  emerged  as  a  most' useful 
one  for  the  study  of  complex  manned  aerospace  systems.  The  model  which 
is  documented  extensively  in  the  literature  (e.g.  Refs.  1  and  2)  is  based 
on  the  assumption  that  the  well- trained ,  well-motivated  human  operator 
behaves  in  a  near  optimal  manner  subject  to  the  extent  to  which  he  under¬ 
stands  the  objectives  of  the  task  and  his  inherent  limitations  and  con¬ 
straints.  The  model  inputs  include  task-related  parameters  (system  dyna¬ 
mics,  disturbance  environment,  available  information)  and  human  operator 
parameters  (perceptual  thresholds  and  time  delays,  human  neuromotor  noise 
observation  noise,  and  the  objectives  of  the  task).  The  latter  can  be  con 
sidered  as  the  input  variables  .of  the  model.  The  output  parameters  of  the 
model  include  measures  of  system  performance,  human  control  activity  and 
various  attentional  characteristics. 

However,  for  a  complete  description  of  the  human  control  behavior  and  its 
impact  on  overall  system  reliability,  it  is  necessary  to  include  the  con¬ 
cept  of  workload  within  the  optimal  control  model  (OCM)  framework  ;  i.e. 
to  assess  in  terms  of  OCM  parameters  how  hard  the  human  controller  has 
to  work  to  achieve  a  given  performance  (criterion).  In  this  context- a 
control  effort  model  has  been  developed  at  NLR  (Ref.  3),  which  will  be 
briefly  reviewed  in  the  following  section. 


Workload  model 

In  the  control  effort  model  (Ref.  3)  it  is  assumed  that  the  human 
control  response  is  partly  determined  by  mechanisms  that  selectively 
tune  the  organism  to  the  stimulus  situation,  by  which  is  meant  both 
selectively  attending  to  some  stimulus  in  preference  to  others  and  in¬ 
vesting  more  or  less  attention  per  source  of  information.  This  can  be 
identified  with  volontary  attention  (Ref.  *0  ,  reflecting  that  the  subject 
attends  to  the  stimulus  because  of  its  relevance  for  performing  the  task 
and  not  only  because  of  its  arousal  function.  Also  involuntary  attention 
is  included  in  the  control  effort  model.  This  can  be  related  to  the  level 
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of  arousal  and  is  largely  dictated  by  the  properties  of  the  displayed 
information . 

The  aspect  of  voluntary  attention  is  incorporated  in  the  model  in  terms 
of  the  overall  level  of  attention,  Pq  (following  reference  6).  The  aspect 
of  involuntary  attention  is  included  in  the  control  effort  model  in  terms 
of  the  sensitivity  of  task  performance  (cost  functional,  j)  to  the  momen¬ 
tary  attention  paid  by  the  subject.  Now,  the  human  operator's  control 
effort  (the  workload  index  E)  is  defined  as  the  ratio  between  the  perfor¬ 
mance  sensitivity  S  and  the  overall  level  of  attention  P  . 

In  formula: 


with 


E  =  S/P  (dB) 
o 


S 


_ 


(dB) 


(D 

(2) 


where  the  partial  derivative  indicates  that  the  other  model  parameters 
are  kept  constant.  (The  cost  functional,  J,  is  the  weighted  stun  of  the 
pertinent  system  variables  ) . 

In  this  way  the  optimal  control  model  has  been  extended  by  a  task  effort 
model  so  that  the  workload  aspects  of  the  task  can  also  be  assessed  with¬ 
in  the  optimal  control  model  framework. 


Predictive  capability 

The  usefulness  of  the  model  to  describe  human  control  dynamics  and 
workload  has  been  supported  in  various  control  tasks  under  laboratory 
conditions,  as  well  as  in  realistic  in-flight  helicopter  control  tasks. 
This  is  summarized  in  the  following  ;  for  an  extensive  presenatation 
the  reader  is  referred  to  ref erences 3  and  5. 

The  experimental  variables  in  the  laboratory  experiment  were  chosen 
so  as  to  include  all  possibly  important  characteristics  of  pilot  behavior 
in  control  situations.  Pilot  equalization  was  experimentally  varied  by 
incorporating  controlled  element  dynamics  of  the  form  k,  k/s  and  k/s2, 
requiring  lag,  "pure  gain",  and  lead  equalization,  respectively. 
Furthermore,  sufficient  variation  in  task  difficulty  was  realized  by 
varying  the  instability  level  of  the  controlled  elements.  For  the  eight 
single-axis  control  tasks  of  the  experimental  program  the  computed  con¬ 
trol  effort  results  were  compared  with  subjective  ratings  (McDonnell 
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rating  ("demand")  scale, see  reference  7)-  The  results,  which  are  shown  in 
figure  1,  indicated  an  excellent  correlation  between  both. 

For  the  in-flight  validation  experiment  (Ref.  5)  an  instrument  hover 
task  and  two  navigation  tasks  on  an  Alouette  III  helicopter  were  chosen. 
The  hover  task  consisted  of  stabilizing  the  helicopter  at  600  ft  alti¬ 
tude  with  minimal  horizontal  (ground)  speed.  The  two  navigation  tasks 
consisted  of  flying  along  a  desired  track  with  an  indicated  air  speed 
of  60  kts  at  a  prescribed  height  :  600  and  150  ft,  respectively.  The 
instrument  information  consisted  of  the  three  attitude  angles  (provided 
by  a  three-axis  ADI),  height  deviation,  horizontal  velocity  components 
(in  case  of  the  hover  task  and  indicated  by  the  flight  director  bars), 
and  cross  track  deviation  (in  case  of  the  navigation  tasks  and  indicated 
by  the  vertical  flight  director  bar). 

The  performance  index  J  was  determined  from  the  recorded  display  para¬ 
meters: 

J\  =  (RMS  h/h  )2  +  (RMS  w/v  )2 

hover  T .  h  h 

L  L 

Jnav  =  (RMS  h/h^)2  +  (RMS  y/y^)2 

Both  performance  indices  incorporate  the  scores  which  were  instructed 
to  be  minimized  :  height  error  (h)  and  horizontal  speed  (v^)  for  the 
hover  task  and  lateral  deviation  (y)  for  the  navigation  tasks.  These 
scores  are  weighted  by  the  corresponding  display  limits  (index  L)  so 
that  J  is  the  sum  of  the  mean-squared  fractions  of  the  full  deviations. 
This  criterion  is  analogous  to  the  cost  functional  of  the  control  model, 
where  it  is  assumed  that  this  criterion  is  minimized  by  the  human  opera¬ 
tor. 

The  comparison  of  the  control  effort  model  predictions  (in  dB)  with  the 
subjective  effort  ratings  of  the  participating  pilots  are  presented  in 
table  .1.  As  the  table  shows,  the  model  predictions  indicate  the  hover 
task  as  the  most  difficult  flight  task,  whereas  both  navigation  tasks 
are  of  the  same  difficulty.  These  predictions  are  thus  in  good  agree¬ 
ment  with  the  results  from  the  subjective  effort  ratings. 

Furthermore,  the  comparison  of  the  model  predictions  of  the  pilot's 
control  behavior  and  task  performance  with  the  experimental  results 
(Ref.  5)  suggests  that  the  model  predictions  represent  the  best  attainable 
performance.  Thus,  the  model  predictions  may  not  be  considered  to  re¬ 
present  the  average  pilot's  control  behavior  but  the  best  pilot's  per¬ 
formance  (which  is  in  agreement  with  the  model  assumptions). 
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A  further  study  (Ref.  5)  showed  that  differences  in  control  behavior 
between  subjects  can  be  "explained"  in  terms  of  two  model  parameters: 
the  indifference  threshold  ratio  (TH)  and  the  overall  level  of  attention 
(P  ).  Both  parameters  reflect  personality  traits,  such  as  control  strategy/ 
motivation. 

In  summary,  the  experimental  results,  obtained  so  far,  indicate  that  the 
model  provides  a  meaningful  representation  of  pilot  workload  and  dynamics 
in  complex  control  tasks . 


PHYSIOLOGICAL  MEASURES 
General 

Despite  a  large  number  of  unsolved  problems  connected  with  the 
reliability  and  validity  of  physiological  measures  (Refs.  8,  9»  10), 
there  are  still  many  indications  that  several  physiological  parameters 
contain  valuable  information  regarding  the  influence  of  the  task  on  the 
human  operator.  As  is  stated  by  Roscoe  (Ref.  8)  :"...  there  is  evidence 
to  support  the  validity  of  using  heart  rate  to  assess  levels  of  work¬ 
load  in  flight  but  it  is  necessary  to  be  aware  of  its  limitations".  The 
following  discusses  some  results  of  NLR  research  on  the  adequateness  of 
several  physiological  measures  to  be  used  as  indicators  of  the  human 
operator's  activation  level. 


Experimental  results 

The  physiological  measures  which  have  been  used  in  NLR  studies  on 

workload  include  : 

-  Heart  rate  parameters  (in  beats  per  minute)  :  mean  (M),  root  mean 
square  (RMS),  standard  deviation  (SD),  root  mean  square  of  successive 
differences  (RMSSD),  and  average  band  power (log  ABP) . 

-  Skin  resistance  parameters  :  resistance  response (SSR)  and  resistance/ 
conductance  level  (SSR/SCL). 

-  Respiration  frequency  (inhalations  per  minute). 

The  measured  values  of  these  parameters  have  been  compared  extensively 
with  the  outcomes  of  subjective  effort  ratings,  performance  indices, 
control  activity  indicators  and  model  workload  predictions.  A  representa¬ 
tive  example  is  given  in  terms  of  the  results  which  were  obtained  for  the 
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helicopter  flying  tasks  mentioned  in  the  foregoing  (see  also  Ref.  11). 

The  tables  2  and  3  show  the  intercorrelations  between  the  measured  values 
of  the  variables  for  the  (relatively  easy)  navigation  tasks  and  the 
(relatively  difficult)  hover  task,  respectively.  These  tables  reflect  a 
very  good  correspondance  between  subjective  effort  ratings  and  the 
physiological  parameters  of  heart  rate  and  respiration.  Especially  res¬ 
piration  frequency  seems  to  be  a  remarkable  sensitive  parameter.  With 
respect  to  the  heart  rate  measures,  the  tables  indicate  that  the  group 
heart  rate  level  and  heart  rate  variability  (RMSSD) ,  is  well  correlated 
with  the  results  from  subjective  ratings.  (Both  heart  rate  measures  can 
be  obtained  relatively  easy).  The  skin  resistance  (conductance)  measures 
seem  less  adequate  to  be  used  as  workload  level  indicators.  Furthermore , 
note  that  the  heart  rate  and  respiration  measures  are  also  well  corre¬ 
lated  with  the  performance  index  J  (mentioned  in  the  foregoing)  and  with 
the  control  activity  indicators  (longitudinal  and  lateral  cyclic,  pedals 
and  collective  pitch,  respectively).  Note,  besides,  that  the  correlation 
between  performance  index  J  and  the  subjective  ratings  are  good  for  the 
relatively  difficult  hover  task  only  (reflecting  near  optimal  control 
behavior  for  this  task). 

Physiological  measurements  have  also  been  performed  in  a  variety  of 
monitoring  tasks  under  laboratory  conditions  (Ref.  12),  thus  imposing 
only  mental  workload  upon  the  subjects  (general  aviation  pilots). 

The  results  clearly  indicated  that  all  heart  rate  measures  (M,  RMS,  SD, 
RMSSD)  were  only  sensitive  to  the  number  of  display  to  be  monitored. 

The  skin  resistance  response  measures  (M,  RMS,  SD,  RMSSD)  and,  to.  a 
less  degree,  the  skin  resistance  level  measures  appeared  to  be  only 
sensitive  to  the  aspect  of  failure  uncertainty  (presence/absence  or 
prior  knowledge  about  failure  type),  thus  reflecting  emotional  aspects 
of  the  task. 

So,  there  is  a  good  support  for  using  heart  rate  measures  (especially 
heart  rate  level  and  heart  rate  variability) ,  as  a  useful  tool  to  study 
pilot  workload  in  control  as  well  as  in  monitoring  tasks  (e.g.  the 
automatic  approach) .  The  usefulness  of  skin  resistance  measures  seems 
to  be  restricted  to  (emotional)  aspects  of  task  uncertainty. 
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CONCLUDING  REMARKS 


Experimental  results  obtained  under  laboratory  and  in-flight  con¬ 
ditions  support  the  usefulness  of  the  optimal  control  model  structure  to 
describe  pilot  control  behavior. 

Furthermore,  the  control  effort  model  predictions  have  been  supported  by 
subjective  ratings  and  physiological  measures.  Although  additional  ex¬ 
perimental  support  for  the  model  is  desirable,  it  seems  to  provide  a 
meaningful  representation  of  pilot  workload  in  complex  control  tasks. 

The  physiological  measures  of  heart  rate  and  respiration  seem  to 
correlate  well  with  subjective  workload  ratings,  overall  performance, 
measures,  control  activity  indicators  and  model  results.  The  skin  resis¬ 
tance  measures  tend  to  be  sensitive  to  uncertainty  aspects  of  the  task, 
more  than  to  performance  aspects. 


REFERENCES 

1  Baron,  S  et  al 


2  Kleinman,  D.L. 
Baron,  S. 

3  Wewerinke,  P.H. 

4  Kahneman,  D. 

5  Wewerinke,  P.H. 

6  Levi son,  W.H. 
et  al 


Application  of  optimal  control  theory  to  the 
prediction  of  human  performance  in  a  complex 
task. 

AFFDL-TR-69-8I ,  March  1970. 

Manned  vehicle  system  analysis  by  means  of 
modern  control  theory. 

NASA  CR-1753,  June  1971. 

Effort  involved  in  single-  and  two-axis 
manual  control  systems . 

NLR  TR  75060  U,  November  1974. 

Attention  and  effort. 

Prentice  -  Hall  Inc.  1973. 

Performance  and  workload  analysis  of  in¬ 
flight  helicopter  missions. 

NLR  MP  77013  -U,  April  1977- 

Studies  of  multivariable  manual  control 
systems.  A  model  for  task  interference. 

NASA  CR-1746,  May  1972. 


85 


7  McDonnell,  J.D. 


Pilot  rating  techniques  for  the  estimation 
and  evaluation  of  handling  qualities. 

AFFDL  TR  68-76,  December  1968. 

8  Various  authors  Assessing  pilot  workload. 

AGARDograph  No.  233,  February  1978. 

9  Heslegrave,  R.J.  et  al  Measuring  baseline-treatment  differences 

in  heart  rate  variability:  variance  versus 
successive  difference  mean  square  and  beats 
per  minute  versus  interbeat  intervals . 
Psychophysiology,  Vol.  16,  No.  2,  1 97 9 • 

10  Wastell,  D.G.  Measuring  heart  rate  variability:  comments 

on  the  successive  difference  mean  square 
statistic . 

Psychophysiology,  Vol.  18,  No.  1,  I98I . 

11  Smit,  J.  An  analysis  of  helicopter  pilot  control 

Wewerinke,  P.H.  behavior  and  workload  during  instrument 

flying  tasks. 

NLR  MP  78003  U,  February  1978. 

12  van  de  Graaff,  R.C.  An  experimental  analysis  of  human  moni¬ 

toring  behavior  in  multivariable  failure 
detection  tasks. 

NLR  TR  81063,  July  1981. 


86 


LA  ^  CO 
ir\  lt\ 


~=J-  -3-  t^- 

t-  VO  VO 


M 

Eh 

P§ 

EH 

« 

W 

o 

o 

> 

• 

H 

C>H 

Lf\ 

EH 

O 

W 

h) 

PQ 

po 

W 

CO 

O  LTV  CO 
LT\ 


LA  4-  t- 
t*—  VO  VO 


CO  ON  VO 
LA  J  J 


Lf\  £■ — 

•  •  • 

t—  VO  VO 


H 

S  « "m 

^  S-S 

o  w 


o 

CT\ 

Q\ 

■ 

• 

• 

LTV 

LTV 

Lf\ 

o\ 

• 

• 

• 

VO 

Lf\ 

-d- 

S  ^  *r 

o  <  <; 

w  s  a 


«  pi  w 

W  I  I 

>  >  > 

o  <  < 

mss 


^  « 

&  5 
<  < 


EH 

<H 

w 

o 

o 

-p 

w 

o 

<D 

<D 

pq 

< 

m 

M 

pD 

ci 

CO 

t-i 

2 

0) 

w 

f> 

< 

87 


(Adapted  from 


Values  of  r  exceeding  .30  are  significant  (p <  .05) 

(Adapted  from  reference  1 1 ) 


On  CVJ 
O  «-  CO 


CO  CO 
CVJ  o 


■*-•*-*- 
C\  4  LA 

CO  4  4 


On  O  ON 

OJ  r- 


**■•*«■ 
4  VO  00 
4  4  t*- 


T-  CO 

o  o 


4  CO 
CVJ  r- 


vo  CO 

o 


O  X 
<h  0  o 
O 

0  g  c 

Ph  ro  *h 


•H  *  •* 

t—  t—  O 
VC  4  b- 


<K-  4*  •*- 

4  VD  ON 

4  4  LA 


O  LAN 
00  O 


00  0J 
00  O 


*r  -¥r 

bD  Ph  t-  LAN 

O  «  4  h  I 

H  3  • 


Q  -K-  -K- 

CO  *-  LAN 

CO  LAN  |t— 


*- 

vd  o 


■¥r 

co  c  co 
t—  o 


LAN  LAN  00 

'4  0J  CVJ 


00  4  C\] 
t-  CO  0J 


co  O  On 
r-  t-  O 


•K-  *K-  -K- 

cvj  co  co 

4-  00  LA 


•K- 

00  ON  CO 
CVJ  t-  CO 


•K-  *r 

t—  o 

VD  CO 


*e 

*r 

t — 

CO 

4t 

l  co 

LAN 

00 

• 

• 

• 

4f  •* 
t—  t — 
I  VD  VD 


•K-  *  -K* 

LAN  r-  00 
VD  -4  VD 


O  t—  VD 
00  O'— 


o  t —  CO 
C—  t—  LAN 


O  r-  00 
t—  VD  C— 


*-  •* 

c—  ON  VD 
4  (M  4 


■H-  *-  *- 

t —  t —  -4 

VD  4  4 


■¥- 

VD 

CVJ 

4 

CVJ 

• 

• 

*- 

VD  00 
0J  4 


00  CO 

O  t- 


0  aJ 
«o  <o 
o  o 


89 


Values  of  r  exceeding  .34  are  significant  (p<  .05) 
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COMPUTED  CONTROL  EFFORT,  E  (o£& ) 

.,  1  A  comparison  of  computed  control  effort  and  subjective  -ratings  for  the  single— axis 
configurations 


RESTRAINT  CONSIDERATIONS  IN  DYNAMIC  ENVIRONMENTS 


Dana  B.  Rogers 
John  ¥.  Frazier 


Air  Force  Aerospace  Medical  Research  Laboratory 
Aerospace  Medical  Division,  Air  Force  Systems  Command 
Wright-Patterson  Air  Force  Base,  Ohio 


ADDRESS  FOR  CORRESPONDENCE:  AFARML/BBS 

Wright-Patterson  AFB,  OH  45433 


ABSTRACT 


The  pilot's  workload  and  ability  to  control  the  aircraft  are 
often  influenced  by  the  restraint  system.  While  the. need  for  restraint 
systems  is  obvious,  criteria  for  good  restraints  are  often  in  conflict. 
For  example,  it  is  desired  to  provide  the  pilot  with  a  system  which 
provides  multi-directional  support  while  also  allowing  unencumbered 
mobility.  Other  considerations  include  comfort,  ease  of  use,  cockpit 
compatibility,  visibility  and  other  factors. 

The  Dynamic  Environment  Simulator  (DES),  a  three-axes  human 
centrifuge,  has  been  used  to  simulate  a  wide  range  of  acceleration 
environments.  An  overview  of  Gx,  Gy,  and  Gz  restraint  techniques, 
problems,  and  tolerance  will  be  presented.  Specific  simulations 
(supported  with  movies  when  available)  of  the  F-1 5  and  KFIR-C2  flat  spin 
environments,  B-1  escape,  and  AFTl/F-16  lateral  maneuvering  will  be 
discussed.  The  pilot's  ability  to  perform  control  inputs  or  initiate 
ejection  sequences  will  be  discussed.  Active  restraints  in  the  form  of 
pneumatic  supports -or  variable  tension  straps  may  be  techniques  for 
providing  improved  restraint  for  unconventional  flight  environment. 
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THE  EFFECTS  OF  SUSTAINED  ACCELERATION,  AIRFRAME  BUFFET, 
AND  AIRCRAFT  FLYING  QUALITIES  ON  TRACKING  PERFORMANCE 


A.  G.  Flranlan 

U.S.  Naval  Air  Development  Center 
Warminster,  Pa. 


Abstract 

To  determine  the  relative  Importance 
of  some  of  the  factors  affecting  the  air- 
to-air  tracking  performance  of  a  pilot 
airframe  system,  U.S.  Navy  test  pilots 
performed  tracking  tasks  in  a  centrifuge- 
mounted  F-4  cockpit  plus  visual  display. 
Sustained  accelerations  up  to  5-0E«  buffet 
intensities  of  up  to  +0.5e»  and  lateral- 
directional  flying  qualities  were  varied 
Independently,  tracking  performance  being 
measured  in  terms  of  miss  distance, 
percentage  time  within  a  fixed  radius  of 
the  target,  and  pilot  opinion  ratings. 
Results  show  a  dominant  influence  of 
flying  qualities,  significant  Influence 
of  sustained  acceleration,  and  negligible 
Influence  of  airframe  buffet. 

Introduction 

At  the  high  speeds  and  angles-of- 
attack  normally  attained  in  air  combat 
maneuvering  flight,  high  sustained  accel¬ 
eration,  airframe  buffet,  and  aircraft 
flying  qualities  combine  to  increase 
significantly  the  difficulty  of  the 
fighter  pilot’s  air-to-air  tracking  task. 
The  literature  concerning  the  relative 
magnitudes  of  these  influences  is  sparse 
and  inconclusive.  Existing  data  on  human 
tolerance  and  performance  in  a  buffeted 
environment  are  limited  to  1-g  flight  and 
are  Inconsistent  due  to  nonuniformity  in 
restraint  systems,  seats,  clothing 
equipment,  control  task  and  scoring 
methods.  'D  Existing  data  concerning 
the  Influence  of  various  levels  of  flying 
qualities  on  tracking  performance  are,  for 
the  most  part,  qualitative  and  do  not 
separate  flying  qualities  effects  from 
buffet  and  g  effects.  All  results  taken 
together  would  seera  to  indicate  that 
flying  qualities  influence  tracking 
ability  considerably,  that  buffet  has  a 
lesser,  though  appreciable  influence,  and 
that  sustained  medium  -to-hlgh  g  has  an 
insignificant  effect.  The  present  study 
was  undertaken  to  determine,  quantitatively, 
just  how  much  of  an  influence  each  of  these 
factors  has.l^) 


Analysis 

Since  these  factors  are  difficult  to 
vary  Independently  in  flight  and  since 


independent  variation  of  them  during  the 
same  tracking  task  was  necessary  for  this 
study,  it  was  decided  to  perform  the  in¬ 
vestigation  in  a  ground-based  simulator. 
The  Naval  Air  Development  Center  human 
centrifuge  flight  simulator  was  parti¬ 
cularly  well  suited  to  the  requirements 
of  the  study,  combining  the  capabilities 
of  a  fixed-base  simulator  with  that  of 
imposing  a  wide  range  of  sustained  accel¬ 
eration  levels. 

Simulator 


The  centrifuge  consists  of  a  10  ft 
diameter  spherical  gondola  supported  in 
a  two  gimbal  system  on  an  arm  of  50  ft 
radius.  The  arm  rotates  counter  clock¬ 
wise,  with  the  gimbals  permitting  the 
gondola  (and,  hence,  the  cockpit  within 
the  gondola)  to  both  pitch  and  roll.  The 
gimbals  permit  orientation  of  the  cockpit 
so  that  the  centrifugal  force  vector,  due 
to  rotation  of  the  arm,  is  coincident  with 
the  resultant  of  the  forces  experienced 
in  actual  flight.  The  cocvpit  was  mounted 
'facing  the  direction  of  arm  rotation, 
resulting  in  the  rotation  of  the  simulator 
pilots  towards  the  hub  of  the  centrifuge 
to  generate  positive  normal  load  factors. 
The  forward  end  of  the  cockpit  was  sup¬ 
ported  on  a  pivot  mounted  to  the  gondola, 
while  the  rear  end  was  supported  by 
lateral  and  vertical  hydraulic  shakers  as 
shown  in  Figure  1. 


The  cockpit  was  an  F-43  Phantom 
cockpit  and  had  been  removed  from  a  strike 
damaged  aircraft.  The  interior  duplicated 
that  of  an  F-4B  except  for  the  instrument 
panel  which  had  been  modified  slightly  for 
a  previous  stall/spin  simulation  program. 
The  flight  controls  were  set  up  to  dupli¬ 
cate  the  force/control  surface  deflection 
characteristics  of  a  current  high  perfor¬ 
mance  fighter,  and  stability  augmentation 
In  all  three  axes  could  be  switched  on  or 
off. 


A  visual  display  was  provided  in  the 
area  bounded  by  the  front  windscreen.  The 
clear  canopy  was  replaced  by  a  dark, 
opaque  one  to  restrict  the  pilot's  visual 
cues  to  within  the  cockpit  (i.e..  Instru¬ 
ments  and  visual  display).  An  all-atti¬ 
tude,  full  color  display  of  earth,  horizon, 
and  sky  >:as  provided  through  a  lens/ 
Attitude  Director  Indicator  (ADI)  system. 
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The  ADI  was  painted  to  provide  a  blue 
sky  with  clouds,  a  distinct  horizon  line, 
and  ground  with  clutter  to  simulate 
populated  areas.  The  zoom  lens  trans¬ 
mitted  the  ADI  display  via  a  mirror  to  a 
projection  screen  in  the  cockpit  wind¬ 
screen,  as  seen  in  Figure  2.  The  lens 
was  used  to  create  a  realistic  sense  of 
altitude  and  altitude  changes. 

Target  and  gunsight  images  were  super¬ 
imposed  on  the  terrain  display  in  a  beam 
splitter.  A  TV  camera  projected  these 
images  from.  a_cathode  ray  tube  onto  the 
beam  splitter.  The  target  was  represented 
by  an  inverted  "T",  and  the  gunsight  was 
a  fixed  reticle  with  horizontal  and  ver¬ 
tical  reference  marks,  as  shown  in  Figure 
3.  The  entire  display  was  viewed  through 
a  fresnel  lens  to  orovide  focus  at 
infinity. 

To  make  the  cockpit  environment  as 
realistic  as  possible,  the  slmlator 
pilots  were  asked  to  wear  full  flight 
gear:  flight  suit,  boots,  helmet,  oxygen 

mask,  torso  harness,  g-suit,  and  full  life 
support  equipment.  The  valve  which 
inflates  the  pilot's  g-sult  in  proportion 
to  normal  acceleration  was  functional  and 
used  during  all  dynamic  runs.  Additionally, 
pressure  breathing  air  was  supplied  to  the 
pilot's  oxygen  mask  from  diving  tanks.  He 
was  able  to  control  cockpit  temperature 
through  air  conditioning  controls  on  the 
right  console. 

Three  Electronic  Associates  Incorporated 
231-R  analog  computers  were  \ised  to  run  the 
simulation.  Aircraft  equations  of  motion, 
control  system  dynamics  and  centrifuge 
coordinate  transformations  were  all  pro¬ 
grammed  on  the  analog  computers. 

Task 

To  provide  the  simulator  pilots  with  as 
realistic  a  task  as  possible  while  slrrul- 
taneoxisly  minimizing  additional  influences 
on  pilot  performance  due  to  the  task 
it'self,  the  target  maneuver  selected  was  a 
bank  into  a  level,  constant  load  factor 
turn ,  a  short  tine  in  the  steady  state 
turn,  and  a  bank  back  to  straight,  level 
flight.  In  terms  of  tine,  the  roll  to  the 
prescribed  bank  angle  took  5  sec,  15  sec 
were  spent  in  the  steady  state  turn,  and 
rollout  back  to  straight  flight  took 
another  5  sec. 

Experience  has  shown  that  air  combat 
maneuvering  is  centered  in  the  transonic 
speed  regime.  The  flight  condition  sim¬ 
ulated  was,  therefore,  i'ach  .8  at  an 
altitude  of  10,000  ft,  with  the  simulated 
aircraft  1500  ft  from  the  target  aircraft 
and  at  his  6  o' block  position  initially. 

The  target  represented  an  aircraft  with  a 
50  ft  wingspan. 

The  ranges  of  buffet  and  sustained 


normal  loads  were  selected  to  encompass 
extremes  encountered  in  flight.  Thus 
buffet  intensity  was  varied  from  0  to 
+>5e  (severe  buffet),  where  buffet  inten¬ 
sity  levels  are  in  terms  of  peak  to  peak 
values,  and  sustained  normal  loads  were 
varied  from  1.3s  to  5.0g.  1-3?  was  selec¬ 

ted  as  a  lower  limit  rather  than  l.Og  to 
permit  continuous  rotation  of  the  centri¬ 
fuge,  while  5.0g  was  found  to  be  an  upper 
physiological  limit  of  continuous  load 
tolerable  for  25  sec  without  grayout. 
Discrete  values  used  within  the  respec¬ 
tive  ranges  were  +. lg  and  +  .25g  buffet, 
and  2.5g  sustained  load. 

Each  task  then  consisted  of  a  random 
riaht  or  left  turn  of  the  target  to  the 
bank  angle  required  to  obtain  1.3,  2.5  or 
5. Og  in  level  flight,  with  buffet  and 
sustained  load  on  the  cockpit  building  in 
the  bank,  remaining  constant  for  15  sec 
In  the  steady  state  turn,  and  decreasing 
again  in  the  final  5  sec  while  the  target 
rolled  back  to  straight,  level  flight.  It’ 
should  be  noted  that  regardless  of  pilot 
stic?<  and  or  throttle  inputs,  the  target 
remained  at  a  constant  1500  ft  range. 

Performance  Criteria 

Tracking  performance  was  measured  in 
terms  of  projected  miss  distance  from  the 
target,  percentage  time  within  10  mils  of 
the  target,  and  pilot  opinion  ratings, 
using  the  Cooper-Harper  scale.  During  the 
simulation,  brush  recorders  printed  out 
time  histories  of  the  aircraft  motion'  var¬ 
iables  as  well  as  weapon/target  lateral 
offset  distance,  vertical  offset  distance, 
and  the  root  mean  square  offset  integrated 
over  5  sec  time  segments.  Pilot  opinion 
ratings  were  given  at  the  conclusion  of 
each  25  sec  task,  ifith  ratings  reflecting 
the  pilot's  assessment  of  the  run  in  terms 
of  his  ability  to  keep  the  pipper  on  or 
return  the  pipper  to  the  target.  All 
pilot  comments  were  recorded  on  tape  during 
the  simulation. 

Pilots 

Eleven  test  pilots  from  the  "'aval  Air 
Test  Center,  Patuxent  Elver,  karyland 
Performed  the  simulation.  All  were  exper¬ 
ienced  fighter  or  attack  pilots,  having 
flown  one  or  more  of  the  various  F-4,  F-8, 
A-’-J-  and/or  A-?  aircraft.  During  the 
simulation,  pilots  were  observed  via  a  low 
light  TV  camera  mounted  in  the  right  side 
of  the  cockpit,  voice  communication  was 
maintained  through  the  "hot"  microphone 
in  the  pilots'  oxygen  masks  and  electro¬ 
cardiogram  signals  'were  continuously 
observed  by  the  medical  monitor. 

Buffet  Phase 

The  simulation  was  divided  into  two 
phases.  In  the  first  phase  the  effects  of 
buffet  and  sustained  normal  acceleration 
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on  tracking  performance  were  measured. 

In  the  second  phase,  the  effects  of  flying 
qualities  were  determined.  To  Isolate 
buffet  and  sustained  acceleration  effects, 
flying  qualities  remained  invariant  in 
the  first  phase  (stability  derivatives 
in  the  six  derree-of -freedom  equations 
of  notion  of  a  rigid  body  were  held  con¬ 
stant)  and  were  the  low  angle -of -attack 
flying  qualities  of  a  current  high  per¬ 
formance  fighter  aircraft. 

Flight  tests  of  current  fighters  with 
cockpit-mounted  accelerometers  show  that 
lateral  and  vertical  component  buffet 
peaks  occur  at  very  nearly  the  sane  fre¬ 
quencies,  with  lateral  intensity  being 
equal  to  or  less  than  the  corresponding 
vertical  intensity.  The  primary  reso¬ 
nance  frequency  excited  in  each  aircraft 
was  that  corresponding  to  the  first  wing 
bending  rode  of  that  aircraft.  Power 
spectral  density  plots  of  the  component 
accelerations  of  the  A-6,  F-104,  F-SA, 
F-5A,  F-10f'A,  and  F-4F  shown  as  functions 
of  frequency  indicate  that  resonance  peaks 
occur  at  frepuenn.ies  between  5  cps  and 
1 8  cps.  (3) (4) (5)  To  limit  the  simulation 
to  a  reasonable  length,  one  representative 
buffet  frequency  was  selected.  Since 
several  of  the  above  aircraft  exhibited 
peaks  in  the  10  cps  area,  that  frequency 
was  used  throughout  the  simulation.’  Fqual 
frequency  and  intensities  were  applied 
vertically  and  laterally. 

To  provide  an  unbiased,  consistent 
data  base,  combinations  of  buffet,  sus¬ 
tained  acceleration,  and  target  turn 
direction  were  randomized  so  that  each 
set  of  12  simulator  runs  encompassed  all 
possible  combinations  of  buffet  and  sus¬ 
tained  acceleration.  Six  pilots  parti¬ 
cipated  in  this  phase  of  the  simulation. 
Bach  pilot  was  scored  for  4  sets  of  runs, 
or  48  total  runs.  Thus  288  scoring  runs 
wore  made  altogether,  with  24  runs  being 
made  for  each  desired  data  ooint. 

Flying  Qualities  Phase 

The  literature  was  researched  to 
determine  fighter  aircraft  flying  qualities 
in  the  higher  arrm  e-of -attack  range  ex¬ 
perienced  in  air  combat.  Angles -of -.at  tack 
up  to  25°  were  considered  for  the  P-4, 

F-5,  F-8,  F-104,  end  F-104.  In  general 
it  was  found  that: 

Longitudinal  flying  qualities 
change  very  little 
Aileron  effectiveness  in  roll 
decreases 

lolling  due  to  yawing  increases 
Directional ■  stability  changes  from, 
positive  to  negative 
Aileron  yaw  changes  from  proverse 
to  adverse 

The  only  fighter  aircraft  showing  a 
noticeable  variation  in  longitudinal 
flying  qualities  at  high  an-les-of -attack 


was  the  F-104.  A  reduction  in  pitch 
damping  was  found  to  occur  as  normal 
acceleration  was  Increased.  This  is 
attributable  to  the  unconventional  "T" 
tail  configuration  of  that  aircraft,  the 
horizontal  tail  being  immersed  in  the  wing- 
fuselage  wake  at  higher  angles-of-attack. 

It  was  concluded  that  the  dominant 
influence  on  aircraft  maneuverability  at 
high  angles -of -attack  was  the  deterior¬ 
ation  in  lateral -d irectional  stability 
and  control.  Serious  problems  noted  for 
several  of  the  above-mentioned  aircraft 
are  wing  rock  and  nose  slice,  where  wing 
rock  is  defined  as  an  undamped  or  di¬ 
vergent  dutch  roll  with  very  large  roll 
excursions,  and  nose  slice  is  a  yawing 
oscillation  caused  by  low  or  negative 
directional  stability.  The  F-4,  in  par¬ 
ticular,  has  been  shown  to  exhibit  a 
progression  of  dutch  roll  from  a  lightly 
damped  rolling  oscillation  (up  to  4  40° 
roll  excursions)  to  an  unstable  yawing 
oscillation  at  23°  angle -of -at tack. ( o) ( ?) 

To  measure  the  influence  of  certain 
lateral -directional  stability  character¬ 
istics  on  tracking  performance,  the 
simulator's  stability  augmentation  system 
was  turned  off,  first,  the  simulator  was 
flown  without  augmentation  to  obtain 
reference  tracking  scores,  and,  finally, 
high  angle -of -attack  lateral -directional 
stability  derivatives  were  set  in  the 
analog  computer.  (Kote  that,  again,  these 
derivatives  remained  constant  regardless 
of  angle-of -attack  commanded  by  the  pilot.) 
Due  to  the  significant  influence  of  aileron 
yaw  on  tracking  performance  demonstrated 
in  tracking  tests  at  NASA  Flight  Research 
Center  and.  at  Calspan  Corporation,  the 
effects  of  proverse,  zero,  and  adverse 
aileron  yaw  were  also  investigated. (- ) 

Before  the  simulator  could  be  flown 
with  stability  augmentation  of f,  adjust- 
gents  had  to  be  made  to  the  yaw  damping 
derivative,  although  it  was  the  actual 
derivative  of  a  current,  high  performance 
fighter  (as  were  all  coefficients).  Dutch 
roll  damping  was  so  light  that  the  target 
could  not  be  kept  within  the  visual  display 
by  the  pilots.  Yaw  damping  was  gradually 
increased  until  the  pilots  indicated 
acceptability ,  raisins-  the  dutch  roll 
damping  ratio  from  .1  to  .31. 

Stability  derivatives  changed  for  the 
high  angle-of -attack  case  were  dihedral 
effect,  roll  damping,  roll  due  to  yaw 
rate,  yaw  due  to  roll  rate  (all  of  whose 
magnitudes  were  increased),  and  direc¬ 
tional  stability  (decreased).  Again  one 
of  the  derivatives  required  modification. 
The  actual  d  irectio>->al  stability  of  the 
fighter  which  was  simulated  decreases  with 
increasing  angle-of -attack,  passing 
through  zero  to  negative  values  above  23° 
Pilots  were  unable  to  track  with  direc¬ 
tional  stability  at  or  below  zero.  The 
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derivative  vras  therefore  increased 
gradually  until  an  acceptable  value  was 
found. 

In  evaluating  the  effects  of  aileron 
yaw,  the  reference  configuration  was 
used,  varying,  only  the  coefficient  of 
roll  due  to  roll  control  from  positive 
to  zero  to  negative. 

To  allow  a  conparison  of  results  with 
those  of  the  first  phase,  sustained 
normal  accelerations  of  1.3,  2.5,  and 
5.0g  were  used.  Two  vibration  levels 
were  Imposed  -  no  buffet  and  +. 25g.  The 
former  allowed  assessment  of  flying 
qualities  influences,  alone,  on  tracking, 
while  the  latter  provided  a  reference 
buffet  condition  for  comparison  with 
the  first  phase  results,  allowing 
assessment  of  tracking  error  increment 
under  the  combined  Influences  of  all 
variables. 

Combinations  of  variables  were  ran¬ 
domized  so  that  a  set  of  6  simulator  runs 
encompassed  all  combinations  without 
bias.  Five  pilots  participated  in  this 
phase  of  the  stimulation,  with  each  pilot 
required  to  perform  3  sets  of  runs  each 
in  the  reference  and  high  angle-of -attack 
configurations,  and  two  sets  of  runs  at 
each  of  the  remitting  roll  control 
conditions,  resulting  in  .60  runs  per 
pilot. 

Results 

Buffet  Phase 

As  noted  previously,  tracking  per¬ 
formance  was  measured  in  terms  of  pro¬ 
jected  miss  distance  in  root  mean  square 
(RMS)  feet,  percentage  time  within  10 
mils  of  the  target,  and  pilot  opinion 
(Cooper-Harper)  ratings.  Mote  that  nil 
is  defined  here  as  one  foot  radial  off¬ 
set  per  thousand  foot  range,  or  since 
range  is  1500  feet,  10  mils  represent 
a  15  foot  error.  RMS  error  was  Inte¬ 
grated  over  5  second  Intervals  of  the 
25  second  task.  Variables  of  Interest 
in  this  phase  were  the  steady  state 
acceleration  and  buffet  levels,  or  the 
middle  15  seconds  of  the  task.  Shown 
in  figure  4  are  the  RMS  scores  of  all 
six  pilots  plotted  versus  normal  accel¬ 
eration  as  a  function  of  buffet  level. 
Since  RMS  was  integrated  over  5  second 
segments  each  pilot’s  scores  were 
averaged,  over  the  15  seconds  at  steady 
state,  and  then  all  pilots'  scores  were 
averaged  together.  Frror  was  recorded 
only  in  those  instances  in  which  the 
target  remained  in  the  visual  display 
over  the  prescribed  time  period.  Some 
runs  were  not  scored  due  to  loss  of  the 
target. 

Figure  4  Indicates  that  sustained 
acceleration  had  a  pronounced  effect  on 


tracking  performance  while  the  effect  of 
buffet  \*as  negligible.  There  is  an 
approximately  linear  Increase  in  error 
with  respect  to  normal  acceleration  of 
6  feet  RMS  per  g,  and  almost  no  change 
in  error  with  respect  to  buffet  intensity 
except  at  5g«  The  scatter  at  5e  Is  due 
to  three  causes: 

1.  Smaller  data  base  due  to  loss 
of  target 

2.  Variation  in  pilot  "learning 
curve"  or  time  to  adapt  to 
the  simulator’s  lack  of  stick 
force  per  g  feel 

3.  The  lack  of  buffet  or  min¬ 
imal  buffet  at  high  g  felt 
unrealistic  to  the  pilot, 
leading  to  decreasing  error 
with  increasing  buffet. 

At  1.3  and  2.5g  the  target  was  virtually 
never  lost,  while  at  5»0g  the  target  was 
lost  80/  of  the  time. 

Figures  5  and  6  re’flect  the  same 
conclusions  as  figure  4.  Figure  5  shows 
percentage  tine  within  10  mils  of  the 
target  versus  sustained  normal  accelera¬ 
tion  as  a  function  of  buffet  level.  While 
percentage  time  is  scattered  over  a  10? 
range  from  no  buffet  to  very  severe 
buffet  at  a  particular  acceleration  level, 
it  decreases  approximately  linearly  with 
increasing  sustained  acceleration  at  a 
rate  of  20;"'  per  g. 

Figure  6  shows  a  barely  perceptible 
increase  in  pilot  opinion  rating  (FOR) 
with  increasing  buffet  intensity,  and 
an  approximately  linear  increase  of  FOR 
with  sustained  normal  acceleration  at  the 
rate  of  .75  per  g.  Up  to  2.5g  the 
aircraft  was  found  to  be  satisfactory, 
thence  going  to  moderately  objectionable 
at  5.0g.  Particularly  noteworthy  is  the 
correspondence  of  FOR  to  RMS  error. 

Figure  7  indicates  a  linear  correspondence 
between  the  two. 

Filot  comments  regarding  this  phase 
of  the  simulation  were  as  follows: 

Simulator  handling  qualities  were 
realistic  except  for  too  great  a 
lateral  sticic  sensitivity 

Lateral  stick  sensitivity  led  to 
mild  PIO’s 

Tracking  field  of  view  was  too 
small 

Lack  of  minimal  buffet  at  high 
sustained  g  felt  unrealistic  and 
abnormal 

Buffet  feel  corresponded  very  well 
to  actual  aircraft  buffet. 

Frlor  to  the  simulation,  pilots 
suspected  buffet  as  being  unimportant  in 
its  direct  effect  on  tracking  performance; 
insofar  as  it  is  a  cue  to  imminent  stall/ 
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departure  it  does.,  however,  divert 
pilot  attention  from  the  target  to  his 
own  aircraft. 

Pilots  displayed  a  high  degree  of 
confidence  In  the  results  of  the 
simulation  -  l.e.,  that  normal  accel¬ 
eration  had  a  more  pronounced  effect 
on  pilot  tracking  ability  than  did 
buffet 


good  at  .5. ^ 


Time  histories  of  the  control  Inputs 
indicated  that  the  pilots  attempted  to 
track  the  target  using  elevator  and 
aileron  control  alone,  until  established 
in  the  steady  state  turn.  Therefore  the 
bank  angle  to  aileron  deflection  transfer 
function  was  considered  for  each  configu¬ 
ration. 


Absolute  quantitative  conclusions 
must  be  qualified  by  noting  that  buffet 
frequency  (10  cps)  was  much  higher  than 
aircraft  response  modes;  at  lower 
frequencies  buff et  might  have  more  of 
an  effect  on  performance. 

Regarding  the  complaints  of  too 
much  lateral  stick  sensitivity.  It 
should  be  noted  that  this  Is  a  defi¬ 
ciency  characteristically  noted  on 
fixed  base  simulators  and  Is  attribu¬ 
table  to  lack  of  roll  rate  cues  to 
stick  Inputs. 

Flying  Qualities  Phase 

Flying  qualities  affected  the 
tracking  solution  primarily  In  the 
Initial  acquisition  of  the  target. 

Once  the  target  was  In  the  steady  state 
turn,  the  tracking  task  became  essentially 
non  maneuvering.  To  properly  assess 
flying  qualities'  Influences,  therefore, 
the  Initial  5  seconds  of  the  task  were 
scored.  A  comparison  was  obtained  by- 
plotting  the  tracking  scores  of  the 
six  pilots  from  the  previous  phase  for 
the  Initial  5  seconds  of  the  task,  as 
shown  in  figure  8.  Since  buffet  and 
acceleration  were  building  to  steady 
state  levels  in  that  segment,  somewhat 
better  performance  was  achieved  than  at 
steady  state. 

From,  the  beginning  of  this  phase  of 
the  simulation  it  was  evident  that 
flying  qualities  would  be  the  factor  of 
paramount  importance  in  tracking  accur¬ 
ately.  The  inability  of  the  pilots  to 
track  with  the  actual  yaw  damping  and 
directional  stability  of  the  simulated 
aircraft  clearly  demonstrated  this. 


For  the  reference  and  high  angle-of- 
attack  configurations  the  dutch  roll  loop 
closures  are  destabilizing.  For  the. 
latter  conf iguration  it  is  seen  that  the 
loop  closure  almost  goes  to  neutral 
stability,  making  it  obvious  why  direction¬ 
al  stability  had  to  be  increased  over  its 
original  value.  At  values  less  than  that 
which  was  used,  the  loop  closure  enters 
the  unstable  half  of  the  plane,  accoimting 
for  pilot  inability  to  track  at  those 
values. 


It  was  expected  that  the  high  angle-, 
of -attack  configuration  would  yield 
poorer  performance  than  the  reference 
aircraft  because  of  the  greater,  difficulty 
In  controlling  bank  anale  with  marginal 
closed  loop  dutch  roll  stability.  Figure 
9  compares  the  tracking  scores  and, 
surprisingly,  shows  that  performance  was 
not  significantly  different.  At  5.0g  the 
high  angle-of-attac?c  flying  qualities  led 
to  an  approximately  10  ft  greater  error, 
while  at  the  lower  g  performance  was 
virtually  the  same.  However,  pilot 
workload  was  higher. 


Consideration  of  the  dutch  roll  loop 
closure  in  the  bank  angle  to  aileron 
deflection  transfer  functions  for  the  • 
various  yaw.  due  to  aileron  control  con¬ 
figurations  reveals  that  bank  angle 
control  is  best  in  the  case  of  zero 
aileron  yaw,  followed  by  adverse  yaw 
and  then  proverse  yaw. 


Figure  10  compares  the  respective 
tracking  scores,  showing  similar  perfor¬ 
mances  with  adverse  and  proverse  yaw.  . 
Clearly  the  neutral  configuration  is 
superior  to  the  latter  two,  tracking 
performance  improving  considerably  - 
l.e.  scores  are  almost  50^  better. 


Interestingly,  the  minimum  dutch  roll 
damping  requirements  of  Pil-F-87853  were 
inadequate  to  permit  satisfactory 
tracking.  With  stability  augmentation 
off,  both  the  spiral  and  roll  modes 
satisfied  level  1  requirements  for  a 
class  IV  aircraft  lfr  flight  phase 
category  A. *5)  Dutch  roll  damping  ratio, 
0.1,  ret  the  level  2  requirements.  The 
pilots  could  not  successfully  track  the 
target  until  the  damping  ratio  had  been 
raised  to  0.3,  a  value  well  above  level 
1  requirements,  .19*  PcDonnell  Douglas 
had  also  found  that  air-to-air  tracking 
was  Impossible  at  a  damping  ratio  of 
,1,  difficult  but  possible  at  .3,  and 


Performance  In  terms  of  percentage 
time  within  10  nils  of  the  target  shows 
the  same  trends  (figure  11).  The  refer¬ 
ence  and  high  angle-of -attack  conditions 
produce  nearly  Identical  results,  proverse 
yaw  is  slightly  better  than  adverse  yaw, 
and  zero  aileron  yawr  is  seen  to  be  a 
greatly  enhancing  quality. 


Pilot  opinion  ratings,  shown  in  figurt 
12,  are  in  slight  disagreement  with  the 
above  results.  In  that  ratings  are 
Identical  for  the  reference  (proverse  yaw) 
condition  and  •.  yzerd  yaw  condition.  Sinc< 
performance  is  better  at  the  latter 
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condition,  the  ratings  may  be  due  to 
either  pilot  fatigue  (since  the  zero 
yaw  condition  was  the  last  set  of  runs 
flown  by  the  pilots)  or  greater  sensiti¬ 
vity  in  pilot  rating  as  the  task  became 
easier.  It  is  easily  seen  that  the  high 
angle -of -attack  condition  required  greater 
compensation  by  the  pilot  than  the  refer¬ 
ence  condition,  although  both  were  flown 
equally  well.  And  the  adverse  yaw  condi¬ 
tion,  although  flown  almost  as  v:ell  as  the 
proverse  yaw  condition,  required  rax i mum 
compensation  by  the  pilot  to  the  point 
where  he  considered  it  a  major  deficiency 
requiring  improvement  at  high  g. 

Comparing  the  reference  condition 
with  that  of  the  augmented,  phase  I 
aircraft,  figures  13-15  show  the 
greater  tracking  accuracy  with  the 
latter  aircraft,  and  the  increased 
pilot  compensation  required  to  attain 
even  the  reduced  accuracy.  With  stability 
augmentation  off,  tracking  error  in¬ 
creased  by  more  than  15  feet  (10  mils) 
or  approximately  100;i,  time  within  10 
mils  of  the  target  decreased  by 
approximately  335”  and  POR's  were  1 
rating  worse.  With  the  modified  dutch 
roll  damping  ratio,  zero  aileron  yaw 
leads  to  performance  very  close  to  that 
with  stability  augmentation  on. 

Sustained  acceleration  effects  on 
tracking  are  the  sane  for  the  stability 
augmentation  on  and  stability  aug¬ 
mentation  off  configurations.  Figures 
13  and  15  show  nearly  the  samp  slopes 
for  both  configurations.  Buffet,  again, 
has  no  significant  effect  on  performance. 

The  paramount  importance  of  flying 
qualities  in  tracking  is  evident.  Not 
only  was  tracking  impossible  at  the 
simulated  aircraft's  actual  yaw  damping 
level  and  directional  stability,  but  even 
with  these  values  raised  to  acceptable 
levels,  tracking  was  poor.  In  terms  of 
R!\S  error,  decreasing  dutch  roll  damping 
ratio  from  .8  (stability  augmentation 
on)  to  .3  degraded  tracking  precision 
as  much  as  a  4  g  increase  in  sustained 
acceleration. 

Pilot  commentary  concerning  this 
phase  of  the  simulation  was  as  follows: 
Tracking  field  of  view  was  too 
small 

Buffet  would  actually  have  more 
of  an  effect  because  of  its 
Indication  of  imminent  stall 
departure 

Buffet  felt  realistic 

Lateral  FIO  was  caused  by  low 
dutch  roll  damping  ratio 

For  equal  proverse  and  adverse 
aileron  yaw,  proverse  is  preferable 


Zero  aileron  yaw  led  to  the  most 
satisfactory  configuration 

Results  of  this  phase  of  the  simula¬ 
tion  are  in  good  agreement  with  those 
of  references  (8)  and  (10).  Reference 
(10)  shows  tracking  ability  to  degrade 
with  decreasing  dutch  roll  damping  ratio, 
tracking  being  impossible  at  a  value  of 
.1.  Tracking  tests  at  NASA  PRC  showed 
that  vring  rock  caused  tracking  errors  of 
almost  30  mils  and  POR's  of  10,  perfor¬ 
mance  significantly  worse  than  that  due 
to  buffet  or  sustained  accelerations  in 
their  tests. 

Considering  the  effects  of  yaw  due 
to  aileron  control,  RASA  FRC  showed 
optimum  tracking  precision  slightly  to 
the  proverse  side  of  zero,  with  pilot 
comments  reflecting  a  preference  for 
proverse  yavr.  Reference  (8)  also 
indicates  a  preference  by  pilots  for 
small  proverse  yavr,  although  optimum 
air-to-ground  weapons  delivery  occurs 
on  the  adverse  side  of  zero.  These  re¬ 
sults  correspond  with  those  of  the  present 
study,  where  optimum  precision  occurs  at 
approximately  zero  yaw  due  to  aileron  and 
where  pilots  prefer  proverse  to  adverse  yaw 

Conclusions 

1.  Buffet  intensities  of  up  to  +.5ff 
at  a  frequency  of  10  cps  have  a  negligible 
Influence  on  tracking  precision. 

2.  Sustained,  high  normal  accelera¬ 
tions  of  up  to  5-Cg  appreciably  degrade 
tracking  precision. 

3.  Flying  qualities  have  the  most 
significant  influence  on  tracking 
precision. 

4.  Dutch  roll  frequency  and  damping 
ratio  have  a  serious  influence  on  tracking 
precision;  tracking  error  increases  as 
either  quantity  decreases,  to  the  point 
where  tracking  becomes  impossible  at 
values  normally  encountered  at  hltrh 
angles -of -attack. 

5.  The  minimum,  dutch  roll  daj.-.cinar 
requirements  of  .'.TL-F-C785R  appear' to* 
be  inadequate  to  accomplish  the  air-to- 
air  tracking  mission. 

6.  For  equal  amounts  of  adverse  or 
proverse  yaw  due  to  lateral  control 
input,  tracking  precision  is  approxi¬ 
mately  the  same,  although  pilots  prefer 
proverse  yavr. 

7.  Providing  a  pilot  with  roll  control 
that  eliminates  yaw  due  to  lateral  control 
input  significantly  enhances  air-to-air 
tracking  capability. 

8.  Tracking  Precision  can  best  be 
improved  by  improving  basic  aircraft 

Stability  and  central  e>t  Mcli  anuUs. 
of -attack. 
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Figure  2.  Combination  of  Terrain  and 
Target  Vist.’al  Display 
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Figure  4.  1'ornal  Acceleration  Effects 
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Figure  3.  Tracking  Disnlay  Arrangement 
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Figure  8.  Normal  Acceleration  Effects- 
Initlal  Segment 
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Figure  10.  TRACKING  SCORES  FOR  VARIOUS 
AILERON  YAW  CONDITIONS 
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Figure  11.  Tracking  Performance  in  Time 
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Figure  9.  Lo  «*•  and  Hi  *<  Tracking  Scores 


Figure  12.  Pilot  Opinion  Ratings 
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REFLECTIONS  ON  THE  EFFECTS  OF  VEHICLE  DYNAMICS 
AND  TASK  DIFFICULTY  ON 
COOPER- HARPER  PILOT  OPINION  RATINGS, 

TASK  PERFORMANCE,  AND  PILOT  WORKLOAD 

by 

Robert  B.  Crcmbie,  Captain,  USAF 
Flight  Dynamics  Laboratory 


This  paper  is  intended  to  set  forth  generalized  principles  cm  a  broad  subject 
which  lies  at  the  heart  of  both  flying  qualities  and  human  factors  research, 
development,  test,  and  evaluation.  These  generalizations  are  the  result  of 
the  author's  study  of  the  flying  qualities,  pilot  workload,  and  pilot  per¬ 
formance  literature.  They  are  submitted  to  the  participants  of  the  Workshop 
on  Flight  Testing  to  Identify  Pilot  Workload  and  Pilot  Dynamics  to  stimulate 
further  discourse  on  the  application  of  these  principles  to  particular  exper¬ 
imental  circumstances  and  to  encourage  the  readers  to  recognize  the  cannon 
threads  linking  the  fields  of  aircraft  flying  qualities  and  human  factors. 
This  paper  first  reviews  and  comments  on  data  reported  by  Piranian  (Refer¬ 
ences  1  and  2)  before  using  these  data  to  illustrate  the  general  principles 
involved. 

The  Experiment.  Almost  ten  years  ago,  Piranian  conducted  an  excellent  ex¬ 
periment  on  the  Naval  Air  Development.  Center  Dynamic  Flight  Simulator  to 
determine  the  relative  importance  of  sane  of  the  factors  affecting  the; 
tracking  performance  of  the  pilot-airframe  system.  The  test  matrix,  sum¬ 
marized  in  Figure  1,  shows  that  the  experiment  was  divided  into  two  phases: 
buffet  and  flying  qualities.  The  results  of  Piranian' s  experiment,  docu¬ 
mented  in  References  1  and  2,  will  be  used  to  illustrate  several  general 
principles  about  the  effects  of  vehicle  dynamics  and  task  difficulty  on 
Cooper-Harper  pilot  opinion  ratings,  task  performance  measures  and  pilot 
workload. 

Buffet  Phase.  The  objective  of  the  buffet  phase  was  to  quantitatively 
determine  the  effects  of  buffet  and  sustained  normal  acceleration  on  air- 
to-air  tracking  performance.  The  flying  qualities  (vehicle  dynamics)  were 
good  (SAS-on)  and  held  constant.  The  piloting  task,  as  stated  in  Reference 
1,  was  to  track  a  target  that  took  five  seconds  to  bank  into  a  steady-state 
level  turn,  maintained  constant  load  factor  (1.3,  2.5,  or  5.0g)  for  15  sec¬ 
onds,  and  returned  to  wings  level  during  the  final  five  seconds  of  the  man¬ 
euver.  Results  in  the  buffet  phase  were  based  on  tracking  performance 
measured  during  the  middle  15  seconds  of  the  maneuver  when  the  piloting  task 
was  simply  to  null  out  the  tracking  error  left  over  from  the  initial  banking 
segment.  The  target  itself  was  not  producing  any  motion  commands  during  the 
steady  state  turn. 

Flying  Qualities  Phase.  The  flying  qualities  phase  of  Piranian 's  experiment 
examined  the  effects  of  degraded  flying  qualities  on  tracking  performance. 
Several  sets  of  lateral-directional  flying  qualities  characteristics  were 
evaluated.  The:  modal  parameters  (extracted  from  Reference  2)  for  each  of 
these  sets  is  shown  in  Table  I  below. 
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TABLE  I 


MODAL  PARAMETERS 

Parameter  SAS-on  Reference  High-ADA  Adverse  & 

_ SAS-off _ Neutral  Yaw 


*d 

.79 

.305 

.408 

.305 

“d 

3.59 

3.24 

2.13 

3.24 

Trm 

.170 

.267 

.180 

.267 

1/T  sm 

.005 

-.025 

-.068 

-.025 

Sp 

.608 

.608 

.608 

.608 

CO 

sp 

5.35 

5.35 

5.35 

5.35 

In  order  to  draw  generalizations  about  the  influence  of  flying  qualities 
(vehicle  dynamics)  on  tracking  performance  and  pilot  workload,  these  sets 
of  flying  qualities  characteristics  will  be  hereinafter  refered  to  as  Good 
(SAS-on) ,  Marginal  (SAS-off) ,  and  Poor  (High-AQA) .  Piranian  also  tested 
"Bad"  vehicle  dynamics  but  the  pilots  were  not  able  to  keep  the  target  within 
the  visual  display  so  tracking  performance  could  not  be  quantitified. 

Tracking  results  reported  for  this  phase  were  rms  tracking  error  and  percent 
time  within  ten  mils  tracking  error  for  the  initial  five  seconds  of  the  man¬ 
euver.  During  that  five  second  period  the  target  banked  (and  increased  its 
load  factor)  to  either  39.7  degrees  (1.3g) ,  66.4  degrees  (2.5g),  or  78.5  de¬ 
grees  (5 . Og)  of  bank  angle.  Therefore  each  sustained  normal  acceleration  level 
reported  by  Piranian  for  this  phase  represents  a  tracking  task  of  corresponding 
difficulty  requiring  the  pilot  to  detect  and  match  the  direction  and  rate  of 
target  motion  as  the  target  performed  a  family  of  rolling  and  pulling  maneuvers. 

Quantitative  Results.  I  have  replotted  Piranian' s  tracking  performance  results 
(Flying  Qualities  Phase)  in  Figures  2  and  3  with  "flying  qualities"  and  "task 
difficulty"  as  the  independent  variables,  respectively.  In  Figure  2  the  gen¬ 
erally  observed  tendency  for  task  performance  to  level-off  even  though  flying 
qualities  (vehicle  dynamics)  degrade  is  evident.  Pilot  ccrnpensat ion ,  reflected 
in  the  average  pilot  ratings,  is  responsible  for  this  leveling-off  tendency. 

Also  suggested  in  this  figure  is  the  catastrophic  drop-off  in  task  performance 
as  flying  qualities  degrade  beyond  the  ability  of  the  pilot  to  compensate  for 
them.  This  drop-off  point,  where  pilot  capabilities  are  saturated,  has  sometimes 
been  referred  to  (e.g.  Reference  3)  as  a  "flying  qualities  cliff". 

Turning  to  Figure  3  we  see  that  task  performance  degrades  as  the  task  beccmes 
more  difficult,  which  is  the  expected  trend.  It  has  been  observed  in  flying 
qualities  experiments  (e.g.  Reference  3)  that  a  difficult  task  is  generally 
required  to  expose  poor  flying  qualities.  The  data  in  Figure  3  confirms  this. 
There  is  a  broad  range  of  task  difficulty  where  task  performance  differentiates 
good  form  marginal  configurations  but  the  poorest  flying  qualities  were  only 
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exposed  using  the  most  difficult  task  tested.  A  reliable  measure  'of  pilot 
workload  (such  as  pilot  opinion  ratings)  is  needed  to  adequately  discriminate 
among  various  levels  of  flying  qualities  at  the  more  moderate  levels  of  task 
difficulty.  As  the  task  becomes  quite  easy  no  change  in  task  performance  may 
be  detected  for  any  level  of  flying  qualities. 

One  may  ask  if  there  is  the  same  "leveling-off"  tendency  in  the  tracking  per¬ 
formance  scores  with  increasing  task  difficulty  as  there  was  for  degrading 
flying  qualities.  The  results  in  Figure  3  are  too  few  to  tell.  However  they 
do  fit  the  pattern  shown  in  data  tcken  from  Reference  4  replotted  in  Figure  4. 

In  that  figure  there  is  an  intermediate  region  of  task  difficulty  where  the 
pilot  seems  to  hold  off  some  of  the  degradation  in  performance  with  task 
difficulty.  At  the  higher  levels  of  task  difficulty,  performance  again  falls 
off  at  a  higher  rate.  Instead  of  "levelling-off"  there  is  actually  only  a 
decrease  in  slope  due  to  pilot  compensation. 

Cooper-Harper  Scale  (Reference  5) .  Which  of  P Iranian's  two  quantitive  track¬ 
ing  performance  measures  is  most  valid?  That  is ,  which  one  relates  better  with 
subjective  pilot  opinion?  To  answer  these  questions  one  must  turn  to  the 
Cooper-Harper  (C-H)  pilot  opinion  ratings  recorded  during  the  experiment.  The 
C-H  pilot  rating  scale,  shewn  in  Figure  5,  is  widely  used  by  flight  test  pilots 
such  as  those  who  participated  in  the  Piranian  experiment.  It  guides  the  pilot 
in  making  qualitative  decisions  about  task  performance  and  pilot  workload  for 
any  compensatory  piloting  task.  Task  performance  is  rated  as  either  "Desired", 
"Adequate",  or  "Not  Adequate".  Pilot  workload,  in  terms  of  pilot  compensation 
required  during  a  compensatory  tracking  task,  can  be  evaluated  as  "not  a  factor", 
"minimal",  "moderate",  "considerable",  "extensive",  or  "maximum  tolerable". 

Figure  6  lists  the  pilot  evaluation  of  task  performance  for  each  configuration 
(inferred  from  average  C-H  ratings)  alongside  the  correspnding  average  tracking 
performance  scores  reported  by  Piranian.  It  is  seen  that  the  measurement  of 
percent  time  within  ten  mils  is  more  consistent  with  pilot  opinion  than  rms 
pipper  error  across  all  levels  of  task  difficulty.  This  tells  us  that  "percent 
time  within  ten  mils"  corresponds  much  better  with  those  aspects  of  performance 
which  were  of  importance  to  those  pilots  in  the  task  they  understood  themselves 
to  be  doing.  Such  a  subjective  validation  process  is  recommended  for  any  pro¬ 
posed  quantitative  performance  or  workload  criterion  using  a  suitable  rating 
scale  and  trained  pilots.  ‘  The  C-H  rating  scale  may  not  be  the  best  to  use 
in  this  validation  process  because  it  links  task  performance  and  pilot  com¬ 
pensation  adjective  pairs  in  a  fixed  hierarchy  designed  especially  for  the 
evaluation  of  airplane  handling  qualities.  One  drawback  of  this  fixed  hier¬ 
archy  is  that  it  does  not  permit  the  pilot  to  rate  performance  and  workload 
independently  (e.g.  "desired"  performance  and  " considerable "  pilot  compen¬ 
sation  is  not  an  allowable  adjective  pair) .  Any  subjective  performance  or 
workload  scale  used  to  validate  performance  or  workload  criteria  ought  to 
give  the  pilot  the  freedom  to  do  this.  Such  a  scale  could  gain  initial 
acceptance  through  being  used  alongside  the  C-H  scale  in  flying  qualities 
experiments  like  Piranian' s.  Once  proven  during  such  compensatory  piloting 
tasks  the  scale  would  then  be  very  useful  for  evaluating  semi-autanated  tasks 
where  pilot  compensation  is  not  the  primary  indicator  of  workload  and  the 
C-H  scale  is  therefore  of  limited  utility. 


One  additional  observation  should  be  made  about  the  Cooper-Harper  scale.  It 
does  not  specifically  address  the  different  components  of  workload  (e.g  emo¬ 
tional  stress,  mental  effort,  multiple  tasks,  cognitive  timesharing) . 

Subjective  workload  scales  that  do  differentiate  among  the  various  components 
of  workload  should  greatly  help  in  the  validation  of  quantitative  workload 
criteria  and  also  aid  the  system  designer.  Sore  such  scales  are  now  under 
development  (e.g.  Subjective  Workload  Assessment  Technique,  Reference  6) . 

Task  Performance  and  Pilot  Workload.  I  have  diagrairmed,  in  Figure  7,  the 
generally-observed  trends  in  task  performance  and  pilot  workload  as  vehicle 
dynamics  or  task  difficulty  are  varied.  Since  each  independent  variable  has 
a  similar  effect  on  both  dependent  variables,  vehicle  dynamics  (for  a  par¬ 
ticular  task)  and  task  difficulty  (for  a  particular  set  of  vehicle  dynamics) 
share  the  abscissa.  Vehicle  dynamics  can  be  expressed  as  aircraft  flying 
qualities  (as  in  Reference  1) ,  system  automation  (Reference  7) ,  or  display 
complexity  (Reference  8) .  Task  difficulty  can  include  not  only  the  task  ob¬ 
jectives  and  initial  conditions  (Reference  4)  but  also  the  external  stressors 
(e.g.  noise,  acceleration)  and  disturbances  (e.g.  vibration,  turbulence)  as 
in  References  1  and  9.  The  scales  of  Figure  7  can  shift  with  respect  to  one 
another,  depending  on  the  circumstances  and  measures  used;  marginal  vehicle- 
dynamics  do  not  necessarily  correspond  to  medium  task  difficulty  and  medium 
pilot  workload  does  not  presume  adequate  task  performance.  There  are,  of 
course,  important  factors  affecting  pilot  workload  and  task  performance,  such  as 
the  typical  variations  in  pilot  physiology,  emotional  state,  bias,  skill  and  tech¬ 
nique,  which  are  not  considered  here. 

Generally  speaking,  then,  there  are  three  regions  of  vehicle  dynamics  or  task 
difficulty  depicted  in  Figure  7.  Region  I  is  where  the  vehicle  is  clearly 
suitable  for  the  task;  performance  is  high  and  workload  is  low.  As  vehicle 
dynamics  worsen  or  task  difficulty  increases,  performance  degrades  steadily 
until  Region  II  is  entered.  Here  the  pilot  actively  maintains  seme  level  of 
performance,  or  at  least  slows  down  the  degradation  of  performance .  Because 
of  this  increased  pilot  compensation,  workload  is  increasing  rapidly.  Where 
the  pilot 1 s  workload  capacity  is  saturated  defines  Region  III .  Here  task  per¬ 
formance  falls  off  a  "cliff."  In  Region  I  the  vehicle  systems  improve  on  what 
the  normal  pilot  could  do  unaided;  Region  II  is  where  the  pilot  oorrpensates 
for  vehicle  deficiencies;  in  Region  III  the  task  may  not  be  performed  safely, 
if  at  all. 

For  vehicle  or  system  specification  purposes,  Region  I  is  more  suitable  for  a 
well-defined  task  performance  spec.  However,  more  conmonly  the  pacing  task  for 
system  design  is  very  challenging  and  usually  falls,  into  Region  II.  To  prevent 
the  vehicle  or  system  from  demanding  too  much  of  the  pilot,  a  workload  specifi¬ 
cation  would  be  preferable  in  that  region.  The  workload  spec  could  define  the 
margin  desired  between  the  pilot  workload  required  and  the  point  where  pilot 
capabilities  are  saturated  (i.e.  the  Region  III  boundary). 

Research  should  be  done  to  define  valid  criteria  to  quantitatively  define  the 
axes  of  Figure  7.  Research  should  then  be  directed  at  defining  the  Region  II 
and  Region  III  boundaries  so  that  adequate  specifications  can  be  written  for 
important  mission  tasks. 


As  performance  and  workload  criteria  are  developed,  experimental  results  should 
be  related  to  the  regions  laid  out  in  Figure  7.  Every  human  factors  or  flying 
qualities  experiment  should  consider  the  interrelationships  between  task  diffi¬ 
culty  and  vehicle  dynamics  in  the  design  of  the  experiment  and  during  the 
interpretation  of  results.  A  better  appreciation  of  these  general  principles 
should  improve  the  value  of  future  research  and  help  the  process  of  flight  test¬ 
ing  to  identify  pilot  workload  and  pilot  dynamics. 

REFERENCES 


1.  Piranian,  A.  G. ,  "The  Effects  of  Sustained  Acceleration,  Airframe  Buffet, 
and  Aircraft  Flying  Qualities  on  Tracking  Performance, "  Proceedings  of  the 
Workshop  on  Flight  Testing  to  Identify  Pilot  Workload  and  Pilot  Dynamics, 
AFFTC-TR-82-5,  Air  Force  Flight  Test  Center,  Edwards  AFB,  California,  (to  be 
published) . 

2.  Piranian,  A.  G. ,  The  Influences  of  Buffet,  Sustained  Normal  Accelerations, 
and  Basic  Aircraft  Flying  Qualities  on  Tracking  Performance  in  Air  Cgnbat 
Maneuvering,  NADC- 73257-30,  Naval  Air  Development  Center,  Warminster,  Pennsyl¬ 
vania,  July  1974. 

3.  Smith,  Rogers  E. ,  "Effects  of  Control  System  Dynamics  on  Fighter  Approach 
and  Landing  Longitudinal  Flying  Qualities, "  Proceedings  of  AFFDL  Flying  Qualities 
Symposium  Held  at  Wright  State  University,  12-15  September  1978,  AFFDL-TR-78- 
171,  Air  Force  Flight  Dynamics  Laboratory,  Wright-Patterson  AFB,  Ohio,  December 
1978. 

4.  Repperger,  D.  W. ,  Rogers,  D.  B. ,  Van  Patten,  R.  E.,  and  Frazier,  J. ,  "A 
Study  of  Task  Difficulty  With  a  Subjective  Rating  Scale,  "  Proceedings  of  the 
Workshop  on  Flight  Testing  to  Identify  Pilot  Workload  and  Pilot  Dynamics, AFFTC- 
TR-82-5,  Air  Force  Flight  Test  Center,  Edwards  AFB,  California,  (to  be  published) . 

5.  Cooper,  George  E.  and  Harper,  Robert  P. ,  Jr.,  The  Use  of  Pilot  Rating  in  the 
Evaluation  of  Aircraft  Handling  Qualities,  NASA  TN  D-5153,  April  1969. 

6.  Reid,  Gary  B. ,  Eggameier,  Thomas,  and  Shingledecker,  Clark  A.,  "Subjective 
Workload  Assessment  Technique, "  Proceedings  of  the  Workshop  on  Flight  Testing 
to  Identify  Pilot  Workload  and  Pilot  Dynamics,  AFFTC-'IR-82-5,  Air  Force  Flight 
Test  Center,  Edwards  AFB,  California,  (to  be  published) . 

7.  Cord,  Thomas  J. ,  IFFC  Flying  Qualities  Simulation,  AIAA-81-1821-CP,  AIAA 
Guidance  and  Control  Conference,  Albuequerque,  New  Mexico,  August  19-21,  1981. 

8.  Korn,  Jonathan,  Gully,  Sol  W. ,  and  Kleinman,  David  L. ,  "Modelling,  Design 
and  Validation  of  an  Advanced  Cockpit  Display  System, "  Proceedings  of  the 
Workshop  on  Flight  Testing  to  Identify  Pilot  Workload  and  Pilot  Dynamics,  AFFTC- 
TR-82-5,  Air  Force  Flight  Test  Center,  Edwards  AFB,  California,  (to  be  published) . 

9.  Nardi,  Louis  U. ,  Kawana,  Harry  Y.,  and  Greek,  David  C.,  Ride  Qualities 
Criteria  Validation/Pilot  Performance  Study-Flight  Test  Results,  NASA  Contractor 
Report  144885,  September  1979. 


106 


TEST  MATRIX 


BUFFET  PHASE 

FLYING  QUALITIES:  STABILITY  AUGMENTATION  SYSTEM  ON 

BUFFET:  0  TO  +.5  GZ  AND  GY  (10  CPS) 

ACCELERATION:  1.3  TO  5  GZ 

6  PILOTS  X  4  SETS  X  12  RUNS  =  2BS  RUNS 
(24  RUNS  PER  DATA  POINT) 

FLYING  QUA1  ITIF.S  PHASE 

FLYING  QUALITIES:  STABILITY  AUGMENTATION  SYSTEM  OFF 

HIGH  -  AOA  STABILITY  DERIVATIVES 

AILERON  YAW:  PROVERSE 
ZERO 
ADVERSE 

BUFFET:  0  AND  +.25  GZ  AND  GY  (10  CPS) 

ACCELERATION:  1.3,  2.5,  5.0  GZ 

5  PILOTS  X  (3  SETS  X  2  F.Q.  +  2  SETS  X  3  A.Y.)  =  60  RUNS 


Figure  1.  Test  Matrix  of  Piranian  Experiment 
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Figure  2. 


Tracking  Performance  Versus  Flying  Qualities 


(Piranian  Data) 
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TRACKING  PERFORMANCE 


Figure  3.  Tracking  Performance  Versus 


TASK  PERFORMANCE  :  MEAN  RMS  ERROR  X  100  (7  SUBJECTS  j  PART 


TASK  DIFFICULTY  :  MEAN  R**2  OF  FORCING  FUNCTION  (  R**2  =  f**2  +  f**2  ) 

o  /  Z  3 


Figure  4.  Tracking  Performance  Versus  Task  Difficulty  (Repperger  Data) 


HANDLING  QUALITIES  RATING  SCALE 


VALIDATION  OF  QUANTITATIVE  PERFORMANCE  MEASURES 


D=  DESIRED  PERFORMANCE  (C-H  1-4) 

A=  ADEQUATE  PERFORMANCE  (C-H  5-6) 

N=  NOT  ADEQUATE  PERFORMANCE  (C-H  7+) 


Figure  6.  Tracking  Performance  With  Pilot  Opinion 
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MEDIUM  HIGH 


or  Too 

TASK  DIFFICULTY:  Low  Medium  High  High 


Figure  7.  Task  Performance  And  Pilot  Workload  Versus  Vehicle  Dynamics 
Or  Task  Difficulty 


AUTOMATED  AIRCRAFT  AND  FLIGHT  CONTROL  SYSTEM  DESIGN 


by 

Morris  A.  Ostgaard 
Air  Force  Flight  Dynamics  Laboratory 
Air  Force  Systems  Command 
United  States  Air  Force 


Automated  aircraft  and  flight  control  design  has  been  a  subject  of  dis¬ 
cussion  and  research  and  development  since  the  early  1900' s.  The  key  issue  at 
this  early  date  was  pilot  workload  and  the  application  of  automation  or  feed¬ 
back  control  to  solve  workload  was  a  subject  of  research  and  experimental 
flying.  The  first  application  of  guidance  and  control  to  aircraft  was  Dr  Elmer 
Sperry's  automatic  stabilizer,  later  called  auto  pilot,  for  which  he  received 
a  50,000  French  franc  prize  in  1914  for  the  most  stable  and  safest  aircraft. 

His  main  purpose  in  developing  this  system  was  to  provide  a  mechanism  to 
relieve  the  pilot  from  the  detailed  task  of  continually  stabilizing  the  vehicle 
and  thereby  making  it  a  useful  machine.  Figure  1  is  a  functional  diagram 
of  this  system  which  illustrates  some  of  the  basic  feedback  principles  involved. 
To  provide  some  background  for  later  discussions,  the  following  will  discuss 
some  of  the  characteristics  of  the  human  operator  and  their  similarity  to 
guidance  and  control  functions  (Reference  1).  The  operator's  characteristics 
as  a  controller  depend  on  four  kinds  of  variables:  control  task  variables, 
which  include  the  system  inputs  and  all  the  system  elements  external  to  the 
operator;  environmental  variables  such  as  ambient  illumination,  temperature, 
vibration,  etc.;  operator-centered  variables  such  as  training,  fatigue  motiva¬ 
tion;  and  procedural  variables  such  as  instructions,  practice,  order  of  pre¬ 
sentation  relating  to  a  given  task.  When  these  variables  are  essentially  time 
stationary  or  invariant  over  an  interval  of  interest,  the  operator  vehicle 
system  can  be  modeled  as  a  quasi-linear  system  much  the  same  as  standard  servo 
loops.  Figure  2  tends  to  illustrate  some  of  these  relationships  showing  that 
the  major  part  of  the  human  operator  problem  is  really  one  of  observability  and 
controllability,  and  the  fact  that  one  of  his  major  tasks  is  the  integration 
of  all  the  information  available  and  attempting  to  determine  the  strategy  or 
control  aspects  that  best  satisfies  the  objective  of  the  mission  task  to  be 
achieved . 

The  fundamental  aspects  of  manual  vehicular  control  theory  can  be  briefly 
summarized  as  follows: 

a.  To  accomplish  guidance  and  control  functions,  such  as  flying 

a  desired  track  in  the  presence  of  disturbances,  maintaining  position  control 
as  in  formation,  terrain  following  or  refueling,  and  flying  intercepts  or 
approaches,  the  human  operator  structures  a  variety  of  closed  loops  about  the 
vehicle  to  achieve  the  control  actions  and  functions  desired  in  terms  of  actual 
vehicle  motions  and  flight  path. 

b.  To  be  satisfactory,  these  closed  loop  systems,  which  include 
both  active  and  passive  elements,  must  share  certain  of  the  dynamic  charac¬ 
teristics  of  a  good  closed-loop  control  and  stablization  system.  As  the 
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adaptive  element  to  accomplish  this  end,  the  pilot  must  make  up  for  any 
dynamic  deficiency  of  the  system  by  appropriate  adjustment  of  its  dynamic 
properties.  This  is  the  terminology  that  was  referenced  as  integration  in 
the  form  of  Figure  2. 

c.  The  cost  of  this  adjustment  is  fundamentally  workload  and  can 
consist  of  stress,  which  includes  vibration,  hazards,  and  environment,  con¬ 
centration  of  operator  faculties  on  the  mission  task  dynamics,  and  the  capa¬ 
bility  to  cope  with  the  unexpected.  This,  then,  illustrates  why  there  is  a 
close  relationship  between  guidance  and  control  and  pilot  workload  in  that 
the  guidance  and  control  systems  can  be  implemented  and  designed  to  reduce 
many  of  the  normal  pilot  operator  functions  as  well  as  defining  ’the  true 
information  or  states  required  for  the  pilot  to  execute  his  mission. 

To  illustrate  some  of  the  trends  and  history  of  guidance  and  control 
over  the  past  few  decades,  Figure  3  illustrates  briefly  what  these  guidance 
and  control  technology  trends  are.  It  summarizes  some  of  the  transitions 
that  have  occurred  over  the  past  two  and  a  half  decades,  beginning  with  the 
1950s  (Reference  2).  But  most  significantly,  it  illustrates  two  basic  trends: 
the  first  is  the  increase  in  guidance  and  control  functions  and  requirements; 
the  second  is  the  progression  from  vehicle  to  weapon  systems  into  what  we  now 
call  air  warfare  systems.  During  the  early  1950s,  the  trend  was  predominately 
vehicle  oriented  in  terms  of  stability  augmentation  systems  to  improve  the 
response  in  control  characteristics.  This  was  followed  by  closed-loop  terrain 
following  systems  to  improve  adverse  weather,  low  altitude  performance  in 
safety  aspects  of  the  system.  Adaptive  system  concepts  became  operational 
in  the  F-lll.  These  concepts,  which  have  been  discussed  extensively  in  other 
publications,  contributed  substantially  to  the  current  modern  control  theory 
and  technology.  As  a  sideline,  a  seldom  recognized  fact  is  that  the  Kalman 
filtering  theory  was  a  direct  outgrowth  of  the  theoretical  work  undertaken 
to  describe,  analyze,  and  design  adaptive  systems. 

Emergence  of  fly-by-wire  technology  in  various  forms  became  evident 
during  the  early  1960s  with  fly-by-wire  spoiler  controls  on  the  F-lll  and 
the  electronic  flight  control  on  the  French  MIRAGE.  Increased  use  of  inertial 
sensors  and  navigation  systems  to  augment  vehicle  of  flight  path  control  for 
reducing  landing  minimums,  and  improve  interaction  with  area  navigation  and 
advanced  air  traffic  control  concepts  during  the  early  1960s,  represents 
initial  steps  in  the  functional  integration  for  increased  performance  and 
workload  reduction.  During  the  mid-1960s,  the  benefits  of  feedback  control 
could  offer  in  alleviating  structural  fatigue  problems  and  turbulence  sensi¬ 
tivity  were  recognized.  This  work  culminated  in  the  application  of  maneuver 
load  control  and  structure  load  control  to  alleviate  turbulence  induced  fatigu 
problems,  and  improve  crew  ride  quality  in  the  B-l  and  other  systems.  Progres 
sing  into  the  early  1970s,  the  MRCA,  the  Swedish  Viegin,  French  Concorde, 
the  F-16,  and  YC-14  designs  witnessed  incorporation  of  sidestick  controllers, 
electronic  displays,  increased  emphasis  on  digital  techniques,  control  and 
propulsion  system  dynamic  interaction,  direct  lift  control,  ride  control,  and 
relaxed  static  stability,  all  of  which  were  made  possible  through  implementa¬ 
tion  of  emerging  technology. 

In  looking  toward  the  future  of  air  warfare  systems,  the  impact  that 
command  and  control  and  communications  will  have  on  the  vehicle  guidance  and 
control  in  terms  of  tactical  control  is  significant.  This  need  generates 
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another  guidance  and  control  dynamics  loop  which  dictates  substantial  dynamic 
and  functional  interaction  to  provide  the  desired  operational  capability.  The 
time-space  positioning  capability  permitted  through  accurate  position  fixing, 
employing  advanced  navigation  systems,  and  the  onboard  vehicle  trajectory 
control  permits  the  implementation  of  the  command  and  control  function  to 
marshal  forces  for  tactical  deployment  and,  also,  offers  a  means  for  redirect 
capabilities . 


The  six-degree-of-freedom  control  projected  through  active  control  tech¬ 
nology  permits  design  freedom  and  tactical  capabilities  unavailable  heretofore 
other  than  in  specialized  rotary  wing  configurations.  These  capabilities  have 
stimulated  application  of  modern  control  theories  involving  strategies,  dif¬ 
ferential  gaming,  to  determine  means  that  can  provide  optimum  trajectories  and 
tactical  options  to  increase  weapon  delivery  accuracy  and  minimize  pilot  work¬ 
load. 


With  this  brief  overview  in  projection,  let's  examine  advances  in  guidance 
and  control  to  determine  how  they  can  be  employed  to  reduce  pilot  workload, 
particularly  in  the  low  altitude,  high-speed  regime.  A  considerable  amount  of 
work  has  been  performed  on  the  application  of  stabilization  systems  to  reduce 
pilot  workload.  However,  the  majority  of  these  systems  have  not  reduced  the 
pilot  workload  to  any  large  degree  because,  in  most  cases,  they  introduced 
unconventional  methods  of  control  which,  in  themselves,  created  a  learning 
problem  (Reference  3).  The  learning  of  these  new  control  techniques  aggrevated, 
in  many  cases,  the  workload  problem.  Also,  many  of  these  developments  concen¬ 
trated  primarily  on  the  control  aspects  ignoring  the  information  display  aspects 
which  provides  the  necessary  information  for  the  pilot  to  act  as  a  closed-loop 
controller . 


The  problem  associated  with  combining  guidance  and  control  systems  or 
automatic  systems  with  pilot  operation  is  the  mechanism  of  pilot  interaction. 
This  is  somewhat  analogous  to  the  cruise  control  on  an  automobile  which  works 
perfectly  for  cruising  on  a  highway;  its  use  in  traffic,  however,  involves  a 
tremendous  amount  of  switching  and  other  unconventional  manipulation.  In  terms 
of  control,  the  first  attempt  to  solve  some  of  these  problems  was  control 
wheel  steering  wherein  a  switching  or  sensor  mechanism  on  the  wheel  or  stick 
was  employed  to  sense  applied  force  by  the  pilot  in  a  normal  manner,  deactivatin 
some  of  the  stabilization  modes  of  the  system  and,  thus,  restore  normal  aircraft 
maneuvering  response.  This  proved  successful  to  a  degree  but  the  fundamental 
control  laws  involved  were  for  stabilization,  and  the  limited  authority  of  the 
system  made  it  extremely  restrictive  and  added  to  the  pilot  workload  rather 
than  easing  it.  A  more  successful  approach  was  the  command  augmentation  system 
design.  This  is  illustrated  in  Figure  4  and  shows  the  requirements  and  charac¬ 
teristics  involved,  the  typical  representation  of  the  system  and,  finally,  the 
basic  issues  involved.  The  command  augmentation  system  provided  a  means  for 
enhancing  the  stabilization  properties  of  the  vehicle  without  deteriorating  the 
maneuvering  performance  while  at  the  same  time  provided  an  input  and  control 
law  capability  that  made  the  vehicle  behave  in  a  manner  that  was  very  desirable 
from  the  handling  qualities  standpoint.  This  was  successful  for  the  traditional 
up-and-away  and  some  precision  tasks  such  as  approach  and  landing;  however,  for 
certain  weapon  delivery  or  mission  functions,  the  general  control  laws  were 
proven  inadequate.  Further  control  analysis  and  flight  research  experience 
indicated  that  an  approach  termed  task-oriented  control  laws  wherein  the  control 
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characteristics  of  the  vehicle  are  deliberately  configured  to  provide  preci¬ 
sion  control  and  performance  for  that  particular  mission  would  be  a  solution. 
These  have  eliminated  many  of  the  problems  of  integrating  the  pilot  with  the 
control  system  and  are  proving  very  successful,  and  indicate  that  the  vehicle 
control  aspect  of  the  problem  is  solvable  employing  guidance  and  control  design 
technology.  At  this  point  it  may  be  advantageous  to  examine  the  role  of 
tactical  air  focus  and  how  this  may  affect  the  design  of  the  flight  control 
system,  its  role  in  system  integration  and  related  areas  of  automation. 

The  primary  role  of  the  tactical  Air  Forces,  because  of  their  mobility 
and  concentrated  fire  power  capability  is  stabilization  of  theatre  conflict 
as  illustrated  in  Figure  5.  In  the  ideal  case  prior  to  ground  force  engage¬ 
ment,  the  tactical  Air  Forces  can  reduce  the  Pact  ground  force  level  to  a 
point  where  ground  forces  alone  can  successfully  defend  and  Contain.  In  a 
minimum  case  for  multiple  engagements,  the  tactical  air  and  ground  forces  can 
reduce  the  Pact  of  ground  force  level  to  a  point  where  tactical  forces  together 
can  successfully  defend.  In  any  event  since  a  cooperative  effort  between  the 
surface  and  Air  Forces  is  necesarry,  secure  communication,  external  threat 
data  and  time-space  positioning  become  vital  elements  for  tactical  battle¬ 
field  control.  Figure  6  through  11  represent  a  scenario  of  events  as  the 
engagement  progresses.  The  key  situation  to  avoid,  as  illustrated  in  Figure  11, 
is  a  breakthrough  which  demands  a  higher  level  of  both  air  and  surface  forces 
to  counter  and  stop  the  primary  exploitation  force.  Since  enemy  defenses  and 
weather  can  have  a  significant  effect  on  the  ability  of  the  tactical  Air  Forces, 
Figure  12  illustrates  projected  Soviet  air  defense  threats  at  the  FEBA.  These 
threats  are  considered  to  be  extremely  mobile  and  can  follow  the  initial  forces 
to  the  FEBA.  If  adverse  weather  conditions  are  super- imposed  upon  this  threat, 
the  only  tactical  option  available  is  a  low  altitude  penetration  and  attack  to 
counter  the  advancing  forces. 

Since  this  role  of  the  tactical  forces  when  combined  with  the  adverse 
weather  and  high  threat  environment  creates  a  high  crew  workload,  a  study  was 
performed  on  the  impact  of  automation  on  crew  effectiveness  and  efficiency. 

Figure  13  illustrates  the  approach  taken  to  assure  a  numerical  methodoloy. 

This  methodology  is  divided  into  two  sections,  a  requirements  section  and  a 
functional/means  section.  One  basic  requirement  is  stabilization  of  theatre 
conflict,  which  is  the  principle  role  of  the  tactical  Air  Force.  Consistent 
with  this  role  is  the  necessity  to  improve  effectiveness  and  avoid  catastrophy. 
Catastrophy  is  defined  as  a  loss  of  aircraft  and  crew  members  due  to  battle 
damage  or  on-board  critical  failure. 

A  means  or  functional  capability  to  achieve  these  ends  or  requirements  are 
illustrated  in  the  second  section,  but  not  necessarily  in  the  order  of  importance 
for  the  various  mission  segments.  This  approach  provides  a  mechanism  whereby 
functional  interactions  during  various  mission  segments  can  be  assigned  numerical 
values  for  determining  areas  of  importance. 

Figure  14  illustrates  a  functional  approach  summary  where  the  previous 
functions  and  means  are  organized  in  terms  of  a  basic  mission  control  function, 
mission  operational  functions  and  mission  support  functions.  The  fundamental 
mission  control  function  is  flight  path  or  trajectory  control,  which  is  the 
only  mechanism  available  for  controlling  the  vehicles  direction  and  magnitude 
of  the  velocity  vector  to  achieve  survivable  penetration  and  timely  terminal 
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conditions  for  the  weapon  delivery.  The  principle  role  of  the  crew  is  that 
of  managing  and  operating  the  mission  operational  functions  to  achieve  the 
mission  control  function.  These  mission  operational  functions  vary  according 
to  the  mission  segment,  as  shown  by  the  ingress  column,  which  includes  flight 
control,  propulsion  control,  navigation,  target  acquisition  and  selection, 
threat  warning,  fire  control,  and  IFFN.  To  provide  this  capability  a  number 
of  mission  support  functions  are  required.  These  are  electrical  distribution 
systems,  hydraulic  distribution  systems,  a  data  distribution  network,  warning 
systems,  and  life  support  which  includes  crew  escape.  This  structure  or 
functional  approach,  indicates  that  the  principle  role  of  the  pilot  or  crew 
is  that  of  functional  and  information  integration  as  opposed  to  vehicle 
control . 

With  this  approach  as  a  functional  baseline,  the  sources  of  pilot  work¬ 
load,  which  is  one  of  the  critical  aspects  of  the  crew  station  was  addressed. 
Figure  15  lists  the  five  major  sources  of  workload,  these  are:  input  satura¬ 
tion,  multi-tasking,  time  line  compression,  pilot  bandwidth  limitations,  and 
fatigue  aspects;  such  as  continuous  low  level  operation.  Using  these  sources 
of  workload,  an  approach  to  define  guidelines  for  which  of  these  functions 
should  be  automated  and  how  should  the  functions  be  integrated  was  formulated. 
Figure  16  illustrates  one  approach  used  in  examining  an  air-to-target  sensing 
and  acquisition  problem.  The  task  performance  requirements,  such  as  time, 
error,  training,  and  crew  loading  are  identified  and  the  various  functional 
crew  aspects  such  as  sensory,  motor,  cognitive,  and  fatigue.  The  descriptors 
for  coding  and  cross  hatching  for  each  of  these  task  performance  requirements 
are  shown  at  the  bottom  of  the  figure,  with  some  guideline  criteria  for  automa¬ 
tion;  such  as,  automate  routine  functions,  automate  memorization  tasks,  automate’ 
precision  operations,  automate  sequentially  timed  operations,  but  do  not  auto¬ 
mate  judgmental  aspects  nor  tactics  as  the  structure  of  the  battle  varies.  To 
provide  some  insight  into  this  particular  area  of  the  mission.  Figure  17  indicates 
some  of  the  crew  interactions  that  can  occur  during  ingress,  engagement  and 
egress  phases  of  an  air-to-ground  mission.  These  are  the  same  means  and  func¬ 
tions  listed  in  Figure  10.  The  numerical  values  at  the  bottom  of  each  metric 
indicates  the  number  of  visual  and  crew  motor  interactions.  The  top  three 
interactions  indentified  are:  threat  warning  and  countermeasures  versus  flight 
control,  propulsion  control  and  flight  control,  and  navigation  and  flight  control, 
with  the  top  six  systems  being  flight  control,  threat  warning,  navigation, 
target  sensing,  external  data  and  weapon  delivery. 

With  this  insight  into  the  number  of  interactions  and  the  level  of  multi¬ 
tasking  that  can  occur  as  a  function  of  the  mission  segments,  Figure  18  sum¬ 
marizes  the  time  line  analysis  for  this  same  air-to-ground  mission.  The  mission 
is  broken  down  into  ten  second  segments  for  ingress  engage,  and  egress  to 
illustrate  the  level  of  simultaneous  tasking  and  the  relative  importance  of 
each  of  these  functions. 

Since  this  multi-tasking  problem  is  emerging  as  the  principle  source  of 
crew  workload,  it  was  necessary  to  consider  what  are  potential  areas  of  auto¬ 
mation  and  what  has  prior  experience  shown  in  this  area.  Figure  19  is  a 
summary  of  candidate  criteria  in  areas  of  automation  based  upon  a  paper  pre¬ 
sented  by  MIT  in  the  early  60' s,  a  recent  literature  survey  and  test  experience. 
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The  commonality  of  these  candidate  criteria  is  not  suprising  and  support 
earlier  guidelines  to  automate  highly  repetitive  functions,  which  are  essen¬ 
tially  regulatory  functions  in  characteristics  and  difficult  functions,  which 
can  be  either  high  order  or  high  bandwidth  functions  and  time  critical  failure 
functions.  TAC  experience  indicates  similar  areas  that  could  be  candidates 
for  automation. 

The  next  series  of  Figures,  20  through  29,  are  technology  oriented  and 
provide  a  description  of  the  functions  listed  in  Figure  14,  and  detailed  dis¬ 
cussion  of  the  current  approach  to  automation,  what  is  an  automation  approach, 
what  are  technologies  needed  for  automation,  what  are  some  current  programs 
that  address  the  needed  technology,  and  finally,  what  are  some  new  thrusts  or 
emphasis  required  to  better  achieve  this  capability.  Since  many  of  these 
charts  are  self  evident,  very  little  discussion  is  required  other  than  to 
note  that  flight  path  control  is  dominant  throughout  all  the  mission  phases 
with  threat  warning  and  countermeasures  as  supporting  technologies  during  the 
ingress,  engagement  and  recovery  phases.  This  technology  breakdown  and  dis¬ 
cussion  of  the  interacting  aspects  of  many  of  these  functions  suggests  there 
are  distinct  areas  of  commonality  between  automation  and  integration  and  a 
role  for  each. 

The  final  phase  of  the  study  activity  was  devoted  to  identifying  a  total 
system  integration  approach  and  relate  automation  to  integration. 

Figure  30  shows  the  structural  integration/automation  approach  which  is 
essential  for  automation  in  combat  aircraft.  Figure  31  indicates  that  high 
levels  of  automation  is  currently  incorporated  in  combat  aircraft  but  limited 
to  the  specific  subsystem  areas;  for  example,  fire  control  is  highly  automated, 
flight  control  is  highly  automated,  many  of  the  weapon  controls  and  threat 
warning  functions  are  highly  automated  including  sensors.  The  most  probable 
reason  for  this  is  that  specific  functional  bottlenecks  are  attacked  as  problems 
occur,  the  logical  extension  of  existing  designs,  the  influence  of  organizational 
alignments  and  limited  cross  technology  integration  technology. 

Because  of  the  cross itechnplogy  integration  limitation,  Figure  32  illus¬ 
trates  that  the  pilot  is  now  the  automation  core  among  the  various  functional 
elements.  He  performs  the  functional  integration  of  information  from  multiple 
sensors  with  the  flight  control,  navigation,  threat  warning,  fire  control, 
propulsion  control,  weapons  and  displays,  and  executes  the  control  functions. 

Since  the  multi-tasking  problem  is  one  of  the  largest  areas  that  can  influence 
crew  workload  and  place  a  practical  outer  limit  on  performance  capabilities, 
a  potential  solution  to  this  problem  is  shown  in  Figure  33.  The  solution  is 
to  re-allocate  functions  by  implementing  a  different  automation  core.  A  logical 
core  is  the  flight  trajectory  and  attitude  control  functions,  because  its  a 
common  function,  is  dominant  in  a  mission  control,  is  central  to  any  air  vehicle 
activity  and  encompasses  several  key  functional  areas,  fire  control,  navigation, 
flight  control  and  propulsion. 

Figure  34  illustrates  a  flow  diagram  from  the  logical  automation  progression 
from  present  day  levels  of  automation;  where  the  crew  integrates  all  the  functions, 
to  the  core  automation  approach;  where  navigation,  flight  control  propulsion, 
and  fire  control  are  the  core  functions;  where  the  crew  only  integrates  the 
threat  weapons  and  sensors  information;  and  finally,  the  ultimate  level  of  auto¬ 
mation;  where  the  crew  operates  as  an  overall  management  function  since  automation 
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and  integration  is  implicit  in  the  overall  design.  Figure  55  depicts  a  con¬ 
ceptual  core  approach  and  development  which  is  essential  to  continued  automa¬ 
tion.  The  input  to  the  core  function  are  sensors,  threats,  external  data  with 
the  output  being  control,  propulsion,  weapons  and  display.  Because  it  is  a 
universal  interface  core,  it’s  a  basis  for  input/output  specification  for  common 
equipments  and  common  software  elements  and  it's  a  logical  growth  to  higher 
automation  levels  as  input  hardware  improves.  The  alternative  to  this  approach 
is  proliferation  of  automation  approach  and  hardware.  This  automation  core 
is  a  flexible  system  structure  involving  logical  partitioning  and  interface 
standards  as  shown  in  Figure  36,  that  can  be  used  as  a  basis  for  developing 
a  prescribed  system  or  implementation  architecture  consisting  of 'distributed 
or  concentrated  redundant  processors,  determination  of  firmware  or  hardware, 
determination  of  various  bus  structures;  and  finally,  standards  applicable 
to  multifunctional  areas. 

The  future  trends  tend  to  indicate  the  need  for  increased  emphasis  on 
precision  flight  path  control,  increased  emphasis  in  flight  control,  propulsion, 
navigation,  and  fire  control  interaction  and  integration  as  a  core  structure; 
and  finally,  the  increase  need  for  a  common  data  distribution  network  as  shown 
in  Figure  38,  which  stresses  integrity  and  capacity.  Figure  39  illustrates 
a  potential  functional  flow  diagram  which  identifies  a  functional  core  to  the 
right  of  the  diagram,  which  encompasses  flight  control,  navigation,  propulsion, 
and  weapon  delivery  as  key  elements  to  flight  path  control.  With  the  inputs 
being  fire  control  based  upon  target  sensing  and  acquisition  information,  threat 
warning  based  upon  information  from  the  on-board  and  external  sources;  and 
finally,  external  data  for  communication,  threat  updates  and  marshalling.  The 
major  challenge  to  achieve  flight  path  or  trajectory  control  for  the  future 
is  the  development  of  a  core  architecture  and  structure  with  the  capacity, 
integrity,  and  flexibility  to  support  these  trends.  Figure  40  depicts,  a 
conceptual  diagram  of  possible  system  design  for  the  future,  indicating  that 
control  theory  and  computation  is  dominant  in  defining  the  system  mathematical 
and  functional  structure,  which  then  can  effectively  be  implemented  in  a  data 
distribution  and  information  processing  system. 
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CURRENT  OEIION  A  RELIABILITY  COMPUTER 

APPLICATION  REQUIREMENTS  CHARACTERIZATION  ANALYSIS  ISSUES  IMPLEMENTATIONS 


FIGURE  4:  AIRCRAFT  CONTROL  SYSTEMS  -  CURRENT  EXPERIMENTAL 
SYSTEMS  AND  PROJECTED 
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AIR/LAND  BATTLE  REQUIREMENTS 
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FIGURE  6  INITIAL  CONTACT 
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FIGURE  13.  ANALYTIC  METHODOLOGY 
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INPUT  SATURATION 
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WORKLOAD  GUIDELINES 
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AIRCRAFT  (CAS) 
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FIGURE  19:  SUMMARY  -  CANDIDATES  CRITERIA  AND  AREAS  FOR  AUTOMATION 


FIGURE  20  FLIGHT  PATH  CONTROL 


FIGURE  21  FLIGHT  PATH  CONTROL 
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FIGURE  22  THREAT  WARNING  AND  COUNTERMEASURES 
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FIGURE  24  TARGET  SENSING  AND  ACQUISITION 
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FIGURE  26  WEAPON  DELIVERY  (Air-Air) 
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FIGURE  27  WEAPON  DELIVERY  (AIR-AIR) 
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FIGURE  28  WEAPON  DELIVERY  (AIR  TO  GROUND) 
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FIGURE  29  WEAPON  DELIVERY  (AIR  TO  GROUND) 
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FIGURE  34:  LOGICAL  AUTOMATION  PROGRESSION 


CO 

a 

u 

< 

a 

OS 

D 

2 

CO 

O 

H 

CO 

a 

CO 

CO 

X 

o 

t-0 

2 

C 

05 

D 

O 

►5 

Eh 

&! 

a 

&! 

2 

O 

< 

CO 

O 

05 

H 

U 

Oh 

s 

Q 

A  A  A 


•  •  • 


X  2 

05  O 

O  H 

Eh  Eh 

U  U 

—  —  — 

W  2 

1  i 

^  D  — 

IS  i< - 

<  Oh  W 

IW  1 

05  Q  OS 

105  1 

Eh  2  W  O 

1  U  1 - > 

<  Q  U 

1  1 

Eh  D  ~ 

2  Eh 

O  H 

H  Eh 

►51  EH 

Oh  rtf 

W 

O  Q 


W 

J 

CO  w 

M 

2  H 

W  05 


W  CO 


O  CO 
Eh  W 
> 

X  o 
Eh  OS 
2  a 


CO 

CO- 

< 

CO 

OS 

H 

05 

Eh 

EH 

D 

a 

05 

2 

»2 

o 

O 

< 

< 

Eh 

< 

O 

O 

o 

a 

•  • 

CO 

W 

Q 

<c 

CO 

a 

►5] 

05 

in 

2 

OS 

Eh 

05 

§ 

s 

< 

< 

m 

W 

X 

►J 

CO 

a 

CO 

o 

o 

u 

s 

CO 

Eh 

<c 

> 

H 

u 

u 

H 

Q 

H 

3 

H 

CO 

a 

OS 

05 

05 

2 

< 

o 

D 

a 

D 

CQ 

1 

1 

a 

nS 

O 

Eh 

H 

X 

a 

a 

• 

• 

• 

155 


CONTINUED  AUTOMATION 


DISTRIBUTED  OR  CONCENTRATED  REDUNDANT  PROCESSORS 


BUSES  OR  BUS  NETWORK 
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DISPLAY 


FIGURE  37;  FUNCTIONAL  INFORMATION  FLOW 
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DIGITAL  COMPUTATION  i  PROCESSED  DATA  SYSTEM 

DATA  NETWORK _ 
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COCKPIT  DESIGM  FOR  THE  FUTURE 


AND 

CHALLENGES  TO  WORKLOAD  MEASUREMENT 
BY 

LARRY  C.  BUTTERBAUGH 
FLIGHT  DYNAMICS  LABORATORY 
AIR  FORCE  WRIGHT  AERONAUTICAL  LABORATORIES 
WRIGHT-PATTERSON  AIR  FORCE  BASE,  OHIO 


INTRODUCTION 


The  implementation  of  current  and  developing  technologies  in  the  area  of 
airborne  subsystem  avionics,  and  control  and  display  hardware,  will  have  a 
dramatic  impact  on  future  cockpit  designs,  crew  workloads,  and  design  and 
evaluation  techniques.  However,  before  speculating  on  the  design  of  future 
cockpits  and  the  workload  problems  they  will  pose,  let's  briefly  overview  the 
last  50  years  of  cockpit  technology  and  crew  workloads. 

The  SPAD  (Fig.  1),  a  World  War  I  fighter,  incorporated  early  display  tech¬ 
nology  for  subsystem  monitoring  which  consisted  of  in-line  gauges.  However, 
with  few  subsystems  to  monitor,  and  skilled  flyers  using  "seat-of-the-pants" 
and  engine  noise/vibration  information  as  indicators  of  system  operation,  work¬ 
loads  associated  with  these  tasks  were  low.  But,  these  low  workloads  were 
probably  replaced  by  other  workloads  associated  with  the  actual  flying  task. 

By  the  time  the  P-26  (Fig.  2)  was  flying  in  the  mid  1930s,  flight  instrumenta¬ 
tion  such  as  airspeed  and  altitude  were  being  displayed  along  with  subsystem 
information.  However,  the  amount  of  information  to  monitor  was  still  fairly 
smal 1 . 

Moving  ahead  to  World  War  II,  the  P-51  (Fig.  3)  had  quite  a  complement  of 
subsystem  controls  and  displays  in  addition  to  attitude  and  navigation  infor¬ 
mation.  And,  of  noteworthy  importance  with  regard  to  the  pilot's  workload, 

,was  the  increased  speed  capability  of  these  aircraft  coupled  with  more  tasks 
to  accomplish.  In  other  words,  the  time  required-time  available  ratio  (a  long¬ 
time  standard  for  workload  assessment)  was  on  the  rise.  An  additional  20-year 
jump  brings  us  to  the  mid  1960s,  and  the  F-lll  (Fig.  4),  which  requires  consid¬ 
erable  monitoring  and  inputs. 

Now  in  retrospect  (Fig.  5),  we  know  that  more  sophisticated  missions  and 
the  inclusion  of  advanced  subsystems  to  assure  completion  of  these  missions 
were  straining  not  only  the  physical  space  of  the  cockpit,  but  also  placing 
extremely  high  task  loadings  on  the  crew.  Fortunately,  however,  display 
and  avionics  technonogy  has  not  been  standing  idle.  Digital  avionics  and 
multi-purpose  displays  have  found  a  place  in  the  cockpit,  as  illustrated  by 
the  Navy's  F - 14  (Fig.  6).  The  incorporation  of  such  displays  allows  more 
efficient  and  effective  display  of  information,  by  displaying  only  what's 
needed,  when  it's  needed.  As  a  result,  workloads  are  able  to  be  held  down  tc 
a  tolerable  level,  simultaneous  to  achieving  desired  system  and  mission  per¬ 
formance.  The  continued  development  of  display  technologies  to  go  with  avi- 
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onics  developments,  therefore,  make  it  very  probable  that  the  next  generation 
fighter  cockpit  may  resemble  that  of  the  Navy's  F-18  (Fig.  7). 

This  introduction  has  quickly  arrived  at  the  state-of-the-art  regarding 
cockpit  technologies.  But,  more  importantly,  we've  witnessed  a  workload 
transformation.  Easily  measured  and  quantified  workloads  associated  with 
task  accomplishment  has  been  replaced  by  the  more  subtle  and  difficult  to 
measure  monitoring  and  decision-making  workloads.  And,  as  will  be  pointed 
out,  the  cockpit  design  problem  will  not  be  getting  any  easier,  especially 
from  the  standpoint  of  advanced  controls  and  displays.  What  is  hoped  for 
is  that  development  of  workload  assessment  techniques  will  be  responsive  to 
the  designer's  needs. 


ADVANCED  COCKPITS 

The  technologies  being  developed  in  the  laboratories  today  will  impact 
future  cockpit  designs  in  many  ways.  Some  of  these  will  include  the  func¬ 
tional  allocation  of  tasks  between  the  man  and  the  machine,  the  modality  of 
subsystems  interface,  advanced  display  hardware,  and  display  formats  and 
symbology. 

Function  Allocation 


The  autopilot  of  current  aircraft  is  one  example  of  computer-aiding  the 
flying  task  by  functionally  allocating  selected  operations  to  be  performed 
automatically.  This  allocation  has  long  been  shown  to  provide  better  tracking 
performance  while  at  the  same  time  reducing,  or  at  least  altering,  the  work¬ 
load  of  the  pilot.  However,  artificial  intelegence  algorithms  (Fig.  8)  may 
one  day  be  put  to  use  in  the  cockpit  to  assist  the  pilot  in  decision-making 
situations. 

Subsystem  Interface 


While  manual  input  and  visually  displayed  output  have  been  the  long¬ 
standing  tradition  of  cockpit  design,  single  alphanumeric  keyboards  that 
replace  the  knobs  and  switches  of  several  seperate  subsystems  are  state-of- 
the-art.  Voice  technology  (Fig.  9),  however,  is  rapidly  developing  into  a 
workable  subsystem  interface  modality.  Studies  have  already  been  conducted 
(Fig.  10)  to  identify  candidate  uses  for  speech  recognition,  as  well  as  speech 
generation  (Fig.  11)  of  advisories,  warnings,  and  status  reports. 

Advanced  Displays 

Underlying  the  potential  application  of  computer  generated  imagery  and 
electronically  projected  images  from  radar,  tv,  and  other  sensors,  is  the 
development  of  advanced  display  hardware.  The  CRT  (Fig.  12)  has  become  a 
feasible  display  device  and  has  been  implemented  in  numerous  cockpits.  Flat- 
panel  devices,  such  as  the  LED  matrix  display,  are  being  developed  as. a 
potential  substitute,  in  certain  cases,  and  have  several  advantages  over  the 
CRT  in  weight,  power  and  depth  requirements. 

Display  Formats/Symbology 

The  most  significant  impact  of  these  display  technologies  is  the  versa- 
tality  of  the  way  the  information  is  presented.  The  first  generation  EADIs 
(Fig.  13)  showed  no  originality  and  simply  repeated  on  a  video  screen  a  two- 
dimensional  picture  of  the  electromechanical  ADI.  There  is,  however,  signi- 

163 


ficant  work  underway  to  study  innovative  pictorial  formats  and  symbology.  The 
following  illustrate  some  of  these  concepts: 

Fig  14  -  A  VSD  format  combining  attitude,  navigation,  and  tactical  situation 
data; 

Fig  15  -  An  HSD  format  showing  navigation  features  and  flight  plan  data; 

Fig  16  -  Another  HSD  format  from  a  higher  altitude  and  showing  the  display 

with  and  without  clutter; 

Fig  17  -  Using  computer-generated  imagery,  a  head-up  view  can  be  projected 
heads-down  to  aid  in  transition; 

Fig  18  -  A  HSD  format  with  range  limitations  overlayed,  to  give  status 
information; 

Fig  19  -  A  MPD  format  with  procedural  information; 

Fig  20  -  A  MPD  format  with  advisory  and  procedural  information. 

DESIGN  TECHNIQUES 

One  can  expect  that  with  the  implementation  of  any  or  all  of  these  cockpit 
technologies,  the  job  of  the  crew  station  designer  will  not  get  any  easier. 

The  laboratory-based  design  process  will  have  to  continue  to  be  mission  oriented 
and  involve  the  user  in  the  definition  and  evaluation  of  the  design.  Further, 
because  of  the  complexity  of  the  systems  and  the  costs  to  simulate  them,  confi¬ 
dent  design  decisions  will  be  necessary  early  in  the  process,  requiring  accurate 
and  reliable  analytic  evaluation  tools.  Also,  the  techniques  applied  in  the 
laboratory  should  be  flight-test  validated,  so  that  laboratory  results  will  be 
confidently  accepted  as  predictive  of  flight-test  and  operational  performance. 

Mission-Based  Design 

Subsystem  and  control/display  definition  based  on  mission  objectives  and 
requirements  is  imperative  (Fig.  21).  The  multiple-mission  concept  implies 
varying  control  and  display  requirements  and  workloads.  To  ensure  the  cockpit 
design  satisfies  all  mission  requirements,  it  is  necessary  to  evaluate  the 
design  with  regard  to  the  total  mission. 

User  Involvement 


The  involvement  of  pilots  in  the  design  process  reaches  beyond  their  role 
in  man-in-the-loop  simulation  and  effectively  begins  with  design  definition 
(Fig.  22).  Pilots  carry  around  expertise  that  can  aid  the  designer  in  under¬ 
standing  what  is  required,  or  desired,  from  the  operational  vantage  point. 

Pilots  can  translate  mission  requirements  into  control  and  display. requirements. 
They  can  evaluate  candidate  designs  and  provide  the  designer  with  valuable 
evaluation  data. 

Quantitative  Techniques  (Fig.  13) 

With  the  costs  of  simulation  and  flight-test  becoming  more  and  more  expen¬ 
sive,  the  value  of  quantitative  design  evaluation  techniques. will  become 
apparent  in  the  design  of  future  cockpits.  Design  alternatives  will  have  to  be 
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evaluated  prior  to  simulation  with  regard  to  a  variety  of  parameters,  preventing 
valuable  simulation  time  from  being  used  for  an  inferior  design.  Some  of  these 
analytic  techniques  exist,  and  others  remain  to  be  developed. 

Flight-Validated  Techniques  (Fig.  24) 

Related  to  the  need  for  quantitative  techniques  applicable  early  in  the 
design  process,  is  the  need  for  laboratory  techniques  -to  be  cross-validated  with 
flight-test  and  operational  data,  in  order  to  ensure  their  sensitivity  and 
accuracy.  The  more  reliable,  accurate,  and  proven  the  techniques  used  in  early 
design,  the  more  confidence  the  designer  will  have  in  the  data  and  his  decisions. 

WORKLOAD  MEASUREMENT  CHALLENGES 
Subjective  Techniques  (Fig.  25) 

With  complex  missions,  workloads  that  are  difficult  to  measure,  and  the 
ability  of  pilots  to  integrate  their  experiences  into  a  determination  of 
workload,  subjective  techniques  are  as  important  in  the  assessment  of  workload 
as  any  quantitative  method.  Subjective  methods  will  continue  to  be  valuable 
tools,  especially  because  of  the  cockpit  trend  towards  more  monitoring  and 
decision-making  tasks  for  the  crew  (the  kinds  of  workloads,  unfortunately,  which 
are  hard  to  assess).  Presently  there  is  need  of  subjective  techniques  which 
produce  ratio-scale  data.  It  will  take  this  type  data  for  meaningful  comparisons 
among  alternative  designs  to  be  possible.  Fortunately,  recent  work  has  been, 
completed  and  current  work  is  underway  which  addresses  this  area  of  subjective 
workload  assessment. 

Measures  and  Criteria 


The  measurement  of  workload,  with  physiological  and/or  psychological 
methods,  will  also  find  an  important  role  in  advanced  cockpits.  One  unique 
opportunity  (Fig.  26)  is  the  use  of  sensitive,  validated  measures  in  an  adapt¬ 
ive  crew  system  concept.  In  this  application,  pertinent  physiological  and/or 
task  performance  parameters  would  be  monitored,  preferably  unobtrusively,  and 
onboard  algorithms  would  determine  what  flight  control  task,  or  other  tasks, 
should  be  transferred  to.  the  machine,  or  onboard  computers,  for  accomplish¬ 
ment.  Such  an  application,  however,  depends  on  the  establishment  of  criteria 
levels  for  the  parameters  monitored.  This  will  not  be  easy,  since  this  has 
been  a  continuing  problem  since  the  inception  of  the  concept  of  workload  and 
the  desire  to  evaluate  it. 


SUMMARY 

In  conclusion,  workload  assessment  appears  to  have  a  major  role  to  play 
both  in  the  design  of  future  cockpits,  and  in  its  application  as  a  component 
of  the  crew  system.  The  challenge  to  the  development  of  workload  measures 
and  predictors  is  the  need  for  sensitive,  diagnostic  techniques  in  the 
design  process,  and  quantifiable,  criterional  measures  for  use  in  the  flight- 
test  and  operational  environments. 


Figure  1.  SPAD 


Figure  3.  P-51 


Figure  4.  F-lll 


Figure  5.  Subsystem  Trends 
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Figure  8.  Computer  Aided  Decision  Making 
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Figure  9.  Voice  Technology 
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Figure  11.  Generation  Task  Candidates 
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NAVIGATION  MODE 

ALTITUDE  >  20,000  FT 


DEFAULT  DATA 


EXAMPLE  DATA  SUPPLEMENTS 


Figure  16.  HSD  Display 
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Figure  17.  Transition  Display 
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1:1  SCALE  -  ZOOM  TO  PROGRESSIVELY 

SMALLER  SCALES 


Figure  19,  Fuel  Endurance 
Procedures  on  MPD  Display 
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Figure  20.  Engine  Fire  Emergency 
Procedures  on  MPD  Display 
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STOL  TACTICAL  TRANSPORT  MISSION 
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Figure  21.  Mission  Scenario  THREAT  ENVIRONMENT 


Figure  24.  Crew  Systems  Design  Process 
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Figure  25.  Subjective  Data  Collecti 
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RELATIONSHIP  OF  THE  FLYING  QUALITIES  SPECIFICATION  TO  TASK  PERFORMANCE 

Frank  L.  George 
David  J.  Moorhouse 

Flying  Qualities  Group,  Flight  Dynamics  Laboratory 
Air  Force  Wright  Aeronautical  Laboratories 

SUMMARY 


The  original  military  specification  was  a  definitive  set  of  flight 
test  maneuver  instructions  intended  to  demonstrate  acceptable  stability 
and  control  (and,  therefore,  handling  qualities)  characteristics.  By  1969, 
the  specification  had  evolved  to  better  fit  the  design  process  by  giving 
acceptable  ranges  of  stability  and  control  parameters,  and  reliability 
goals.  It  appears  to  some  observers  that,  by  this  same  time,  the  design 
process  had  been  segmented  so  there  was  a  polarization  between  the  aero¬ 
dynamics  and  performance  people  on  one  side  and  the  control  designers  on 
the  other.  As  a  result,  there  seems  to  be  a  competition  -  the  aerodynamicists 
challenging  the  control  people  with  increasingly  exotic  configurations  and 
larger  performance  envelopes;  the  flight  control  designers  countering  with 
more  complex  control  laws  and  feedback  structures.  This  fragmented 
approach  leads  the  designers  away  from  the  essence  of  our  flying  qualities 
definition  -  the  eventual  integration  of  pilot  and  airplane  characteristics 
for  some  useful  task.  Thus  it  seems  capability  is  sometimes  put  into  an 
airplane  just  because  it's  possible  rather  than  because  it's  useful. 

Of  course,  the  need  to  properly  interface  the  pilot  and  airplane  was 
recognized  long  ago.  Development  of  the  understanding  and  tools  to 
accomplish  the  feat  has  been,  and  continues  to  be,  slow  and  difficult. 

Notable  developments  have  been  the  servo  and  optimal  pilot  models,  the 
Cooper-Harper  and  related  subjective  evaluation  scales  and  extensive  com¬ 
putational  aids  for  dynamic  analysis  and  data  processing.  Numerous  design 
methods  and  criteria  have  been  proposed  and  tested  using  these  basic  results, 
only  a  sampling  of  which  will  be  discussed  in  this  paper.  Interest  in  pilot 
identification  with  regard  to  flying  qualities  arises  because  the  use  of 
experimental  methods  in  criteria  development  as  well  as  in  design  and  eval¬ 
uation  leads  to  the  need  for  isolating  the  pilot  component  of  close'd  loop 
responses  in  order  to  define  vehicle  requirements  or  dynamics. 


SPECIFICATION  BACKGROUND 


The  subject  of  flying  qualities  has  been  discussed  under  one  title  or 
another  for  as  long  as  people  have  tried  to  make  the  airplane  a  useful 
device.  Figure  1  traces  the  early  evolution  of  military  requirements  for 
what  we  now  call  flying  qualities.  The  quote  from  the  1907  contract  for  a 
Wright  flyer  was  essentially  the  entire  specification  of  "flying  qualities". 
The  human  piloting  aspects  were  recognized,  however,  through  the  statement 
"It  should  be  sufficiently  simple  in  its  construction  and  operation  to  permit 
an  intelligent  man  to  become  proficient  in  its  use  within  a  reasonable 
length  of  time."  The  first  formal  specification,  AAF-C-1815,  was  essentially 
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a  collection  of  required  flight  test  maneuver  instructions  to  demonstrate 
stability  and  control  limits.  An  example  of  the  requirements  is  shown  in 
Figure  2.  As  can  be  seen  from  the  maneuver  description,  this  requirement 
was  as  much  a  demonstration  of  performance  capability  as  it  was  control, 
and  the  only  consideration  for  the  pilot  was  the  limit  on  stick  force. 

Since  Class  III  airplanes  in  that  specification  included  fighters,  attack 
planes  and  dive  bombers  some  relationship  can  be  seen  between  the  pullup 
requirements  and  mission  needs,  but  the  demonstration  maneuver  doesn't 
necessarily  represent  the  way  a  pilot  would  use  the  pullup  capability 
operationally. 

Reference  1  traces  the  development  of  appreciation  for  the  pilot's 
influence  on  the  controlled  airplane's  dynamic  properties,  and  of  analytic 
methods  to  describe  and  study  the  pilot-airplane  system.  (Ron  Anderson's 
paper  at  this  workshop  also  gives  a  detailed  discussion  of  pilot  models) . 

A  significant  change  occurred  with  the  introduction  of  pilot-vehicle 
analysis  methods  and  the  use  of  rating  scales  to  quantify  the  pilot's 
subjective  assessment  of  flying  qualities.  What  followed  was  a  period  of 
specialization  in  flying  qualities  research  along  two  main  tracks  which,  at  times 
have  even  appeared  to  diverge.  One  track  emphasized  the  interaction  between 
the  pilot  dynamics  and  airplane  dynamics ;  pilot  models  were  developed  more 
as  a  research  activity  with  little  practical  acceptance.  The  other  track 
concentrated  on  defining  open  loop  characteristics  which  would  satisfy 
closed  loop  requirements.  Augmentation  concepts  were  developed  to  provide 
reasonable  stability  and  effective  control  over  larger  and  larger  performance 
envelopes . 

What  was  the  effect  on  requirements  and  specifications?  A'Harrah  and 
Woodcock  (Ref  2)  assert  the  current  military  specification  requirements 
(Ref  3)  "are  largely  predicted  on  his  [the  pilot's]  need  to  perform  fine 
tracking".  They  further  assert  the  specification  has  evolved  from  a  demon¬ 
stration  document  to  one  primarily  useful  as  a  design  guide.  Many  require¬ 
ments  in  MIL-F-8785B  were  stated  in  terms  of  open  loop  modal  characteristics. 
Application  of  such  requirements  in  design  could  imply  a  much  greater 
reliance  oh  analytic  methods  to  evaluate  or  "demonstrate"  flying  qualities. 

In  practice,  however,  t[ie  approach  has  often  been  to  use  these  methods  to 
provide  a  configuration, which  is  finalized  using  piloted  simulation.  Reference 
4  claims  the  results  of  this  pragmatic  approach  have  been  less  than 
successful. 

The  emphasis  on  precision  tracking  is  compatible  with  the  commonly  used 
linear,  small  perturbation  analysis  techniques.  However,  we  must  remember 
there  are  large  amplitude  maneuvers  that  also  require  precise  pilot  control. 
Examples  are  target  acquisition  and  precise  heading  changes.  Such  maneuvers 
require  different  considerations  of  pilot  dynamics  as  well  as  analytic 
capability  for  nonlinear,  coupled  dynamic  modes. 

Flying  qualities  requirements  on  airplane  dynamics  based  on  task  needs 
are  exemplified  by  the  limits  placed  on  the  time  delay  of  airplane  response 
to  pilot  command  (Ref  3,  para.  3.5.3).  The  time  delay  defined  here  is  the 
effective  delay  -  that  sensed  by  the  pilot.  The  pilot's  ability  to  achieve 
acceptable  precision  tracking  is  quite  sensitive  to  this  delay  and  the 
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numerical  requirement  of  no  more  than  0.1  second  delay  for  Level  I  flying 
qualities  is  based  on  experimental  validation  of  values  necessary  to  main¬ 
tain  a  pilot  rating  of  3.5  or  better,  as  the  delay  increases.  (A  Cooper- 
Harper  rating  of  3.5  or  better  constitutes  Level  I  flying  qualities.) 

Figure  3  illustrates  the  impact  of  time  delay  on  closed  loop  characteristics 
when  the  pilot  is  included.  As  the  figure  shows,  the  stability  reduction 
due  to  the  pilot's  0.3  sec  delay  dominates  the  closed  loop  characteristics 
for  wgp>2  rad/sec,  and  in  effect  limits  the  bandwidth.  The  open  loop 
requirements,  on  the  other  hand,  tend  to  minimize  phase  shift  and  extend  the 
bandwidth. 


FLYING  QUALITIES  CONSIDERATIONS  IN  DESIGN 


At  this  point  in  our  discussion  of  flying  qualities,  specification 
and  pilot  modeling  relationships  it  might  be  well  to  review  the  general 
definition  of  flying  qualities,  shown  below: 

Flying  Qualities:  Those  stability  and  dynamic  response 
characteristics  of  an  airplane  and  its  control  system 
which  impact  the  pilot's  ability  to  complete  some  useful 
task  or  mission. 

As  you  can  see  this  very  general  definition  of  flying  qualities  involves  the 
dynamic  properties  of  the  airplane  as  the  pilot  sees  it  -  that  is,  with  all 
augmentation  and  control  dynamics  (in  both  normal  and  failed  states) .  The 
second  important  point  is  the  relationship  to  a  mission  event  or  task.  In 
other  words,  the  design  goal  of  achieving  "constant  flying  qualities"  across 
the  Operational  Envelope  does  not  necessarily  mean  "constant  dynamics". 
Finally,  <-;te  purpose  of  the  machine  is  to  assist  or  enable  the  human  operator 
to  accomplish  some  job.  Hence,  the  characteristics  of  the  airplane  should 
not  make;  it  an  encumbrance.  With  this  definition  in  mind,  let  us  now 
consider  how  flying  qualities  enters  the  design  process. 

Figure  4  depicts  the  system  design  problem  as  a  hierarchy  of  decision 
making  which  begins  with  identifying  some  need  and  determining  how  well  it 
must  be  satisfied.  The  system  problem  becomes  a  design  problem  only  if  the 
required  capability  does  not  exist.  At  this  point  detailed  requirements 
are  needed  to  guide  the  design  of  a  new  or  modified  system.  It  is  at  this 
point  that  flying  qualities  requirements  are  usually  considered  to  appear. 
However,  availability  of  general  task  performance  related  requirements  at 
the  higher  decision-making  level  could  favorably  impact  system  design 
decisions.  A  couple  of  general  comments  about  this  hierarchy  are  in  order 
before  continuing.  First,  note  it  is  not  a  one-way  path;  iterations  can 
occur  at  and  between  all  levels.  Second,  the  decision  hierarchy  is  very 
general  and  can  be  repeated  at  each  level. 

To  illustrate  the  way  in  which  flying  qualities  enters  the  design 
problem,  let  us  consider  the  block  diagram  of  Figure  5.  This  diagram 
clearly  shows  the  importance  to  flying  qualities  requirements  of  the  inter¬ 
face  between  the  human  and  the  machine.  It  further  illustrates  a  difficulty 
in  specifying  the  flying  qualities  requirements  for  the.  airplane.  That  is, 
the  pilot's  perception  of  the  vehicle's  characteristics  is  strongly  in¬ 
fluenced  by  his  environment.  As  the  figure  shows,  this  environment  involves 
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aspects  of  the  cockpit  displays,  the  vehicle  dynamics  and  task  requirements. 
The  dynamic  nature  of  this  interface  underscores  the  idea  that  flying 
qualities  is  not  a  constant  in  the  system  design.  As  Figure  6  depicts, 
definition  of  the  requirements  as  well  as  the  flying  qualities  is  a  dynamic 
part  of  the  design  process. 

In  it's  earlier  forms  the  specification  could  have  accommodated  the 
application  of  pilot  modeling.  Performance  in  the  required  flight  test 
maneuvers  could  readily  have  been  calculated  using  an  appropriate  form  of 
pilot  model.  Note  that  model  forms  for  open  loop  commands  may  not  correspond 
to  tracking  models.  Performance  standards  and  pilot  limitations  would  then 
have  completed  flying  qualities  requirements  suitable  for  both  analytical  and 
flight  test  demonstration  compliance.  This  is  speculation  and  there  are  many 
arguments  against  that  approach.  It  is  also  academic  since,  as  we  have  seen, 
the  military  specification  developed  along  the  lines  of  open  loop  modal  param¬ 
eters.  We  need,  therefore  to  consider  the  current  and  future  direction  of  the 
requirements . 

The  implication  (strong  but  not  intended)  that  MIL-F-8785B  forced  design 
toward  classical  dominant  mode  characteristics  has  been  corrected  in  the  "C" 
revision.  The  same  requirements  now  apply  to  equivalent  parameters  represent¬ 
ing  the  overall  airplane  response  to  pilot  input,  not  to  a  particular  mode  of 
the  system.  Figure  7,  from  Reference  5,  illustrates  the  use  of  equivalent 
parameters  to  demonstrate  compliance  with  requirements.  In  this  example  the 
actual  system  is  modeled  using  a  classical  short  period  equivalent  system. 

As  can  be  seen,  some  questions  remain  to  be  answered  regarding  interpretation. 
The  backup  document  also  discusses  alternate  criteria  including  closed  loop 
criteria.  Thus,  the  specification  applies  to  designs  that  can  be  represented 
by  modal  characteristics  and  allows  others  to  meet  alternate  requirements. 

This  approach  is  being  continued  and  expanded  in  the  latest  revision  effort. 
Drafts  of  the  proposed  MIL-Standard  and  Handbook  are  currently  being  dis¬ 
tributed  for  critique.  For  the  pitch  axis,  equivalent  system  parameters 
shall  meet  requirements  as  in  MIL-F-8785C.  Plus,  an  alternate  is  stated  in 
terms  of  bandwidth  of  the  (open  loop)  pitch  attitude  response.  This  is 
intended  to  address  the  closed  loop  piloting  requirements  more  directly,  in 
a  form  that  is  amenable  to  the  design  process  and  also  flight  test  evaluation. 
We  intend  to  continue  developing  the  requirements  in  this  direction. 


THE  USE  OF  PILOT  DYNAMIC  MODELS 


Analytic  methods  to  describe  the  pilot’s  behavior  evolved  because  of  the 
long  recognized  need  to  account  for  the  pilot's  effect  on  aircraft  responses. 
Greater  use  of  dynamic  analysis  during  design  has  resulted  in  refinement  of 
pilot  modeling  techniques  in  both  the  classical  servo  and  optimal  control 
approaches.  Emphasis  on  the  task  relationship  to  flying  qualities  has 
created  some  interest  in  greater  use  of  pilot  modeling  in  defining  flying 
qualities  requirements.  However,  the  whole  premise  behind  current  pilot 
modeling  and  its  relationship  to  flying  qualities  is  that  closed  loop  tracking 
is  the  most  demanding  flying  qualities  test.  With  the  advent  of  missiles, 
computing  sights,  trainable  guns  and  improved  sensors  this  may  no  longer  be 
a  valid  assumption.  The  pilot  may  spend  much  less  time  tracking,  and  the 
tracking  he  does  may  require  less  precision.  The  implication  of  this  change 
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in  the  pilot’s  role  is  that  the  servo  and  optimal  control  tracking  models 
will  no  longer  be  adequate  by  themselves.  There  will  be  a  need  to  consider 
decision  making  and  workload  along  with  intermittent  control  activity. 

Further,  identification  methods  based  on  single  axis  small  perturbation 
approaches  will  not  be  appropriate.  To  provide  a  baseline  for  these  future 
considerations,  let's  consider  where  we  are  with  respect  to  pilot  modeling 
methods  as  applied  to  flying  qualities  requirements. 

Applications  of  pilot  modeling  to  flying  qualities  criteria  development 
have  been  based  on  the  classical  or  optimal  control  tracking  models,  and 
have  followed  two  tracks.  One,  use  knowledge  of  the  pilot  dynamics  to  define 
vehicle  characteristics  that  will  result  in  acceptable  closed  loop  tracking 
responses.  Two,  use  knowledge  of  the  pilot  model  parameters  to  predict  or 
correlate  with  the  pilot's  subjective  assessment  of  the  airplane  character¬ 
istics  (i.e.,  Cooper-Harper  Rating).  Neal  and  Smith  (Ref  6)  used  crossover 
pilot-vehicle  analysis  to  propose  criteria  for  pitch  attitude  dynamics,  as 
shown  in  Figure  8.  Chalk  (Ref  7)  used  whis  work  as  the  basis  for  a  proposed 
new  pitch  dynamics  requirement  for  MIL-F-8785B,  Figure  9.  Chalk's  intent  was 
to  define  requirements  for  the  augmented  open  loop  airplane  dynamics  consider¬ 
ing  the  pilot-aircraft  closed  loop  properties.  More  recently,  Hoh  (Ref  8) 
has  proposed  tentative  specification  requirements  using  a  definition  of  the 
vehicle  frequency  response  bandwidth.  Figure  10.  His  contention  is  that 
bandwidth  is  a  fundamental  parameter  for  correlating  vehicle  dynamics  with 
pilot  acceptance  for  some  particular  task.  The  common  element  in  these 
efforts  has  been  to  specify  airplane  characteristics  in  terms  of  frequency 
response  parameters. 

Another  approach  to  flying  qualities  criteria,  specification  in  terms 
of  task  performance,  was  proposed  by  Qnstott  (Ref  9).  This  approach  parame¬ 
terized  the  flying  qualities  problem  from  the  viewpoint  that  task  performance 
was  primary  and  airplane  dynamics  should  be  adjusted  to  fit  an  assumed  pilot 
capability  to  get  desirable  task  performance.  In  contrast  to  the  previously 
described  criteria,  Onstott  defined  requirements  in  the  time  domain.  As  Figure 
11  shows,  Onstott  defined  his  criterion  in  terms  of  closed  loop  task  performance 
measures,  rms  error  during  target  acquisition  and  time- on- target  during  a 
finite  period  of  gunsight  tracking.  The  pilot  rating  boundaries  shown  were 
generated  empirically  and  appear  logical  in  a  heuristic  sense.  That  is,  small 
rms  error  and  long  time- on- tar get  should  warrent  a  good  pilot  rating  -  unless 
excessive  workload  is  required  to  achieve  the  performance.  Further  analysis  of 
the  pilot  parameters  would  address  the  workload  question.  Also,  the  data 
presented  here  are  for  pitch  axis  tracking.  The  more  realistic  multiaxis  case 
should  be  investigated  to  fully  evaluate  this  approach  to  defining  criteria. 

The  second  track  in  applying  pilot  model  analysis  to  flying  qualities 
was  initiated  by  Anderson  (Ref  10).  His  technique,  called  paper  pilot,  was 
based  on  the  parameters  of  the  servo  pilot  model.  Subsequently,  Hess  (Ref  11) 
Schmidt  (Ref  12)  and  Levison  (Ref  13)  have  pursued  rating  prediction  methods 
based  on  the  optimal  pilot  model.  Hess  proposed  scaling  the  quadratic  per¬ 
formance  index  of  the  optimal  model  to  get  a  Cooper-Harper  rating  prediction. 
Schmidt  further  investigated  Hess'  approach  in  an  effort  to  develop  an 
analytic  control  synthesis  methodology  which  would  specifically  address 
flying  qualities  considerations.  Levison  proposed  a  rating  prediction  method 
similar  to  Anderson's  but  using  elements  of  the  optimal  control  model. 
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Specifically,  he  suggested  rating  expressions  which  are  weighted  sums  of  the 
probability  of  task  related  variables  exceeding  certain  limits  and  the  pilot's 
attention  level;  in  other  words,  a  combination  of  performance  and  workload 
factors. 

Now,  finally,  how  does  all  this  relate  to  pilot  identification?  First, 
of  course,  is  the  need  to  validate  pilot  models  with  experimental  data.  Both 
the  servo  and  optimal  control  models  have  been  defined  based  on  tracking 
experiment  data.  Furthermore,  application  of  the  optimal  model  requires 
specifying  values  for  a  large  number  of  parameters  for  which  experimental  data 
provides  baseline  values.  Second,  experimental  evaluation  of  flying  qualities 
for  mission-related  tasks  leads  to  the  need  for  identifying  the  pilot  com¬ 
ponent  of  closed  loop  responses  in  order  to  verify  the  airplane  dynamics. 
Identification  of  the  pilot's  dynamics  is  difficult  under  ideal  conditions 
because  of  the  inherent  time  delays,  remnant  and  potential  time-varying 
behavior.  In  the  flight  test  environment  the  task  is  made  even  more  difficult 
by  the  problem  of  defining  the  input  signals  to  the  pilot.  For  example, 
standard  correlation  or  spectral  analysis  methods  require  a  known  random¬ 
appearing  external  forcing  function.  Elkind  (Ref  14),  Wingrove  (Refs  15,  16) 
and  Heffley  (Ref  17)  have  proposed  alternative  methods  based  on  measurements 
normally  available.  Elkind  compared  power  spectral  identification  methods 
with  a  method  based  on  orthonormal  functions.  Wingrove  showed  how  the  pilot's 
time  delay  could  be  used  to  advantage  in  estimating  pilot  dynamics  when  the 
random  forcing  function  is  not  available.  Heffley  and  Jewell  adapted 
Wingrave's  idea  to  devise  an  "on-line"  method  for  generating  coefficients  of 
an  assumed  pilot  equation  from  a  closed  loop  task. 

A  different  viewpoint  toward  pilot  identification  than  that  expressed 
by  most  people  arises  from  the  use  of  pilot  opinion  rating  and  associated 
comments  to  both  validate  flying  qualities  criteria  and  evaluate  airplane 
flying  qualities.  This  reliance  on  the  pilot's  subjective  rating  stimulated 
the  efforts  mentioned  previously  to  predict  pilot  ratings  using  model  param¬ 
eters  and  measures  of  task  performance.  Smith  and  Torgerson  (Ref  18)  con¬ 
sidered  the  Cooper-Harper  rating  to  be  a  vector  and  investigated  methods  to 
identify  the  factors  and  states  the  pilot  includes  in  his  rating  vector. 

Their  preliminary  results  indicated  about  six  parameters  dominate  his 
evaluation  but  more  data  are  needed  to  statistically  isolate  them  from  a 
larger  set.  Identification  of  these  states  would  provide  another  means  of 
correlating  pilot  acceptance  with  airplane  characteristics. 

One  of  the  most  difficult  elements  of  the  pilot's  characteristics  to 
quantify  is  workload.  The  difficulty,  of  course,  is  because  of  the  indirect 
way  workload  enters  the  problem.  Reference  19  discusses  several  concepts  of 
workload  as  well  as  a  number  of  metrics.  However,  no  clear  conclusions  or 
recommendations  emerge.  The  importance  of  workload  to  rating  prediction 
methods,  task  performance  oriented  criteria  and  the  increasing  complexity  of 
the  pilot's  job  as  a  system  manager  all  point  to  a  need  for  a  better  capa¬ 
bility  to  account  for  this  aspect  of  the  pilot's  characteristics. 


CONCLUSIONS 


We  have  shown  that  flying  qualities  was  recognized  very  early  in  the 
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history  of  manned  flight  to  be  a  dominant  consideration  in  the  dynamic  pilot- 
airplane  interface.  As  a  formal  entity,  flying  qualities  initially  spun  off 
from  stability  and  control,  and  performance  considerations.  Many  requirements 
in  the  military  flying  qualities  specification  are  still  presented  in  terms  of 
classical  airplane  modal  dynamic  parameters.  As  a  consequence  of  the  impression 
that  these  modal  requirements  would  not  apply  to  highly  augmented  aircraft,  the 
specification  has  been  ignored  or  circumvented  in  some  cases.  The  reasoning  in 
these  cases  has  been  that  high  authority  stability  and  flight  control  systems 
would  alleviate  any  potential  problem  (e.g.,  prevent  high  angle-of-attack 
problems  by  limiting  control  stick  authority) .  Efforts  have  been  made  in  the 
current  modifications  to  the  military  specification  to  clarify  and  improve  the 
applicability  of  requirements  to  anticipated  highly  augmented  aircraft,  (e.g., 
equivalent  system  discussion  and  alternate  "bandwidth"  requirements) . 

It  is  recognized,  however,  that  mission  requirements  will  play  an  even 
greater  role  in  future  military  aircraft  design.  As  these  requirements  becomie 
more  stringent,  the  effective  integration  of  aerodynamics,  performance  and 
control  will  be  essential  to  meeting  them.  Specification  of  open  loop  air¬ 
plane  characteristics  based  on  implied  or  indirect  closed  loop  requirements 
will  not  be  adequate.  It  appears  knowledge  of  the  pilot's  dynamic  and  perfor¬ 
mance  characteristics  is  necessary  to  quantitatively  relate  airplane  dynamic 
properties  to  many  task-oriented  design  requirements.  Many  such  requirements 
are  closed  loop  in  that  they  involve  the  pilot  as  an  active  controller. 

Others  involve  the  pilot  intermittently  as  a  controller  or  decision  maker. 

The  Flying  Qualities  Group  has  an  effort  planned  to  develop  closed  loop 
criteria  to  consider  future  piloting  requirements  more  directly.  Both  the 
development  of  these  requirements  and  their  application  will  require  identi¬ 
fication  and  analysis  of  the  pilot's  dynamics  and  workload. 
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E-Io.  Longitudinal  Control; 


E-Io(I)(a).  Longitudinal  Control  at  High  Speeds.  -  All 
Class  III  airplanes  shall  demonstrate  their  ability  to  recover  from 

HIGH  MACH-NUMBER  DIVES.  THIS  TEST  SHALL  BE  MADE  BY  DIVING  VERTICALLY 
15.000  FEET  FROM  SERVICE  CEILING  AT- WHICH  TIME  PULL  OUT  WILL  BE 
STARTED  EXCEPT  IN  NO  CASE  WILL  PULL  OUT  BE  STARTED  BELOW  15,000  FEET. 

Airplanes  in  this  cuss  shall  demonstrate  their  ability  to  recover 

FROM  SUCH  DIVES  AT  NORMAL  ACCELERATIONS  OF  AT  LEAST  3G  BY  THE  APPLI¬ 
CATION  OF  A  STICK  FORCE  NOT  EXCEEDING  150  POUNDS  OR  THROUGH  THE 
USE  OF  ANY  OTHER  RECOVERY  DEVICE. 


FIGURE  2.  EXAMPLE  OF  EARLY  FLYING  QUALITIES  REQUIREMENT 
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FIGURE  A.  THE  SYSTEM  DESIGN  PROBLEM  -  A  DECISION  HIERARCHY 


PILOT-AIRPLANE  CLOSED  LOOP  SYSTEM 


FIGURE  5. 


PILOT-AIRPLANE  FLYING  QUALITIES  INTERFACE 
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CLOSED  LOOP  FLYING  QUALITIES  ANALYSIS 


FIGURE  6.  DYNAMIC  INTERACTION  OF  CRITERIA  IN  DESIGN  PROCESS 


APPLICATION 


DEFINITION 


FIGURE  7.  DEMONSTRATING  COMPLIANCE  WITH  EQUIVALENT  PARAMETERS 
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FIGURE  8.  NEAL-SMITH  PITCH  DYNAMICS  CRITERIA 


3.2.2  Longitudinal  Maneuvering  characteristics 

flight.  The  frequency  response 


3.2. 2.1  Pitch  dynamic  response  in  Maneuvering 
of  pitch  attitude  to  elevator  control  force  sh. 

(AA/Af )#  and  A4  t  are  within  the  liMits  of  figure  1,  using  the  following 
reference  frequency  (*J0) . 


Levels  1  and  2; 


Flight  Phase 
Category 


*>#•»  rad/sec 


A 

1,  IV 

3.0 

11.  Ill 

2.0 

1 

ALL 

1.2 

C 

ALL 

1.2 

Use  •  1.1  rad/sec  for  all  Flight  Phase  Categories  and 
Classes. 

This  requirement  applies  for  responses  of  any  Magnitude  that  Might  be 
experienced  in  service  use.  In  addition,  the  contractor  shall  show  that 
the  alrplene  has  acceptable  response  characteristics  in  atmospheric  distur¬ 
bances. 


FIGURE  9.  CHALK'S  PROPOSED  PITCH  DYNAMICS  REQUIREMENT 
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FIGURE  10,  HOH'S  PROPOSED  BANDWIDTH  CRITERION 
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FIGURE  11.  ONSTOTT'S  PITCH  TARGET  TRACKING  CRITERION 
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DEVELOPMENT  OF  CONTROLLER  REQUIREMENTS 
FOR  UNCOUPLED  AIRCRAFT  MOTION 


J.  Hodgkinson,  K.  D.  Citurs  (McDonnell  Aircraft) 
R.  H.  Hoh,  (Systems  Technology,  Inc.) 

T.  J.  Cord,  (AFWAL) 


The  use  of  uncoupled,  six-degree-of-freedom  (6  DOF)  motion  is 
rapidly  becoming  state  of  the  art  in  terms  of  necessary  flight  control 
laws  and  aerodynamic  capability.  McDonnell  Aircraft  and  Systems 
Technology  Inc.  are  under  contract  to  AFWAL  to  develop  design  criteria 
for  cockpit  control  devices  for  6  DOF  motion  which  will  assure  compati¬ 
bility  among  the  pilot,  the  control  device(s)  and  aircraft  response.  This 
will  allow  efficient  implementation  of  the  6  DOF  control  capability  and 
improved  mission  performance.  The  effort  includes  a  study  of  existing  data 
followed  by  a  ground-based  simulation.  In  the  simulation  planning, 
appropriate  methods  for  estimating  the  impact  of  6  DOF  capability  on  pilot 
activity,  will  be  defined.  The  estimate  will  be  in  terms  of  pilot  workload, 
controller  activity  and  display  requirements.  It  will  be  used  to  explore  the 
tradeoffs  of  manual  vs  automatic  control  for  those  aircraft  classes  and  tasks 
where  6  DOF  capability  is  of  benefit. 

The  presentation  will  be  a  progress  report  briefly  describing  the  study 
of  existing  data,  and  outlining  the  simulation  plan.  Emphasis  will  be  placed 
on  the  plans  for  pilot  workload  estimation. 


A  FLIGHT  RESEARCH  VIEWPOINT 


Donald  T.  Berry  and  Terrence  W.  Rezek 

My  comments  will  be  brief  since  much  of  what  I  had  in  mind  can  be 
summed  up  by  endorsing  what  has  already  been  said,  particularly  in 
regard  to  the  papers  by  Col.  Milan,  Col.  O'Donnel,  and  Ron  Anderson. 

However,  from  the  point  of  view  of  a  flying  qualities  flight  research 
engineer  and  a  human  factors  engineer  with  a  strong  involvement  in  the 
pilot-vehicle  interface,  my  remarks  can  be  categorized  under  three 
headings: 

1.  Needs  for  Flight  Research 

2.  Needs  o£  Flight  Research 

3.  Needs  of  Flight  Test 

Not  all  the  items  I  will  touch  on  are  pilot  dynamics  or  workload 
items  in  themselves,  but  they  all  are  strongly  related  in  that  pilot 
dynamics  and  workload  measurements  would  greatly  enhance  the  research  I 
will  discuss.  With  respect  to  the  first  item,  needs  for  Flight  Research, 

I  will  discuss  three  subtopics: 

la.  In-Flight  measures  of  pilot  strategy  and  gain 

lb.  Simulation  validation  for  critical  tasks 

lc.  Parametric  data  on  pilot  controllers 

In  regard  to  item  la: 

In  the  fall  of  1980  a  workshop  on  pilot  induced  oscilations  was 
held  at  Dryden.  One  of  the  main  conclusions  of  the  workshops  was  that 
existing  pilot  in  the  loop  analysis  techniques  were  highly  successful  in 
identifying  generally  PIO  prone  or  PIO  resistant  configurations.  However, 
to  apply  these  techniques  as  accurate  design  tools,  more  data  was  needed 
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on  pilot  gain  and  control  strategy  as  a  function  of  vehicle  class, 
dynamics  and  task.  Flight  measurement  of  these  characteristics  is 
needed  because  pilot  gain  and  strategy  is  often  different  in  a  ground 
simulator  than  it  is  in  flight.  Which  leads  us  to  item  lb,  "Simulator 
Validation  for  Critical  tasks." 

An  area  of  concern  is  the  difference  between  real  world  flight  and 
simulation  with  respect  to  pilot  behavior  in  response  to  the  stress  of 
critical  situations.  For  most  situations,  particularly  involving  routine 
flight  management  tasks,  flight  and  ground  simulator  results  compare 
very  well.  However,  for  many  critical  tasks,  such  as  final'  phases  of 
landing,  abrupt  collision  avoidance,  aerial  refuel  trig1  etc. ,  significant 
differences  are  observed  between  pilot  technique  and  behavior  in  flight 

i  .  «■  ,<  t  )'  ■ -i' lir¬ 

as  compared  to  simulator  results.  An  example  of  this  is  illustrated  by 

figure  1,  which  compares  ground  and  flight  results  for  pitch  PIO  tenden¬ 
cies  of  an  advanced  aerospace  vehicle.  The  ground  simulator  was  the 
Anies  FSAA  (Flight  Simulator  for  Advanced  Aircraft)  arid  the  airborne 
simulator  was  the  Air  Force/Calspan  TIFS  (Total  In-Flight  Simulation). 

The  horizontal  axis  is  the  PIO  rating  scale  number  as-signed  by  the 
pilots.  A  rating  of  1  indicates  no  PIO  tendency,  and  a  rating  of  6,  an 
uncontrollable  PIO.  It  can  be  seen  that  there  is  as  significant  differ¬ 
ence  in  PIO  tendencies  between  ground  and  airborne  results.  Because  of 
the  heavy  reliance  on  ground  simulation  in  the  design  and  development 
process,  many  advanced  prototype  aircraft  have  been  found  to  have  serious 
deficiencies  in  the  .human  factors  and  displays  area.  The  cost  of  redesign, 
modification,  and  flight  test  to  alleviate  such  problems  is  extremely 
high.  It  would  be  highly  productive  then,  to  determine  the  limits  of 
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applicability  of  ground  simulators  for  these  situations,  and  strive  to 
obtain  a  basic  understanding  of  the  differences  between  flight  and 
ground  results  so  that  they  might  be  eliminated  or  ameliorated  in  the 
future. 

A  basic  tool  in  this  process  would  certainly  be  pilot  dynamics  and 
workload  measurements  for  identical  scenarios  in  flight  and  in  ground 
based  simulators. 

Pilot  controller  is  the  third  area  recommended  for  flight  research. 
Despite  some  excellent  work  in  this  field,  pilot  controllers  have  been 
designed  on  an  ad  hoc  basis,  and  nobody  seems  really  satisfied  with  the 
results.  A  systematic  investigation  of  controller  characteristics  for  a 
reasonable  range  of  parameters  does  not  exist.  This  is  especially  true 
for  sidestick  controllers.  And,  because  of  the  sensitivity  of  controllers 
to  g,  stress,  pilot  cues,  motion,  etc.,  a  flight  validated  data  base  is 
needed.  And,  of  course,  good  measures  of  pilot  workload  are  needed  to 
implement  such  research. 

The  second  category  to  be  discussed  are  the  needs  o£  flight  research. 
That  is,  those  items  that  can  contribute  to  the  increased  effectiveness 
and  efficiency  of  human  factors  flight  research.  Three  subtopics  will 
be  discussed  here,  namely: 

2a.  Non-intrusive  measurement  techniques  and  compact 
lightweight  instrumentation 

2b.  Remotely  Piloted  Vehicle  (RPV)  utilization  in  human 
factors  research 

2c.  More  communication  and  interaction  between  flying 
qualities  engineers  and  human  factors  engineers 
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With  respect  to  item  2a  we  are  often  faced  with  a  dilemma.  In 
order  to  determine  what  a  pilot  is  doing  in  a  particular  task,  we  often 
have  to  intrude  with  measurements  or  procedures  that  interfere  with  the 
task  at  hand.  Consequently  the  task  we  want  to  study  is  altered!  A 
need  exists  for  non-intrusive  means  of  making  the  desired  measures  that 
do  not  alter  the  task  at  hand  or  encumber  the  pilot  with  unrealistic 
procedures.  There  is  also  a  need  for  research  equipment  compatible  with 
the  confines  of  a  fighter  cockpit.  A  case  in  point  is  the  oculometer. 
Presently,  these  systems  are  bulky  and  heavy  and  require  elaborate 
calibration  procedures.  This  negates  their  routine  use  in  flight  test 
except  in  large  aircraft  under  limited  circumstances.  Efforts  are 
underway  to  develop  more  practical  oculometer  systems  and  these  are  to 
be  applauded.  It  is  hoped  that  similar  effort  can  be  applied  to  other 
equipment.  Many  advanced  aircraft  are  flown  routinely  at  the  Dryden 
Flight  Research  Facility.  If  they  could  be  routinely  instrumented  to 
measure  pilot  scan  patterns,  workload,  etc.,  a  large  data  base  for  a 
variety  of  tasks  and  vehicle  configurations  could  be  gathered  at  low 
cost. 


Remotely  piloted  research  vehicles,  (RPRVs)  are  being  used  to 
conduct  highrisk  flight  research  or  operations.  It  has  been  observed 
that  RPRV  missions  generate  high  pilot  gains  during  critical  flight 
conditions.  The  motivation  and  stress  levels  produced  during  RPRV 
operations  approach,  and  can  occasionally  surpass,  those  encountered 
during  onboard  operations.  Such  levels  can  not  be  duplicated  on  even 
the  most  expensive  simulator. 
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In  addition,  input/output  devices  (individual  displays,  control 
mechanisms,  etc.)  or  entire  systems  may  be  changed  in  the  ground  cockpit 
with  greater  ease  and  less  expense  and  personal  risk  than  in  a  fully 
human  rated  aircraft.  The  computers  that  reside  with  RPRV  systems 
easily  accommodate  a  variety  of  signal  manipulations  for  information 
transfer  (displays),  control  modification,  or  data  acquisition.  RPRVs 
then,  offer  a  cost  effective  way  of  bridging  the  gap  between  the  labora¬ 
tory  and  the  real  world.  Figure  2  illustrates  the  RPRV  technique  as 
applied  to  the  highly  maneuverable  aircraft  technology  (HiMAT)  vehicle. 

The  interaction  between  allied  disciplines  such  as  flying  qualities 
and  human  factors  is  becoming  more  active,  but  a  case  can  be  made  for 
some  crosstraining  to  achieve  a  greater  appreciation  of  each  other's 
methodology.  For  example,  human  factors  engineers  tend  to  use  large 
numbers  of  subjects  and  rely  on  statistical  inference.  Flying  qualities 
engineers  tend  to  use  a  small  number  of  highly  expert  well  calibrated 
subjects  and  rely  on  pilot  commentary  and  subjective  rating  scales. 
Combinations  of  these  techniques  may  be  the  best  way  to  approach  pilot 
modeling  and  workload  measurement. 

Needs  of  Flight  Test 

The  third  and  final  area  I  would  like  to  touch  on  relates  to  the 
contributions  human  factors  and  display  technology  can  make  to  flight 
testing  itself.  This  consideration  may  be  somewhat  peripheral  to  the 
main  theme  of  this  workshop,  but  is  sufficiently  related  to  warrant 
consideration. 
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For  the  most  part,  pilot  dynamics  and  workload  is  thought  of  in 
the  context  of  operational  missions  such  as  air  combat,  approach  and 
landing  etc.  We  are  all  familair  with  the  complex  tasks  associated  with 
various  mission  phases.  However,  a  flight  test  is  a  mission  also,  and 
although  it  has  many  tasks  in  common  with  other  missions,  in  many  cases 
the  requirements  are  unique.  For  example,  in  NASA  flight  research,  it 
is  often  required  to  precisely  match  wind  tunnel  test  conditions  and 
flight  conditions  so  that  accurate  correlations  can  be  made.  Extremely 
close  tolerances  on  several  variables  may  be  required  to  be  held  simul¬ 
taneously  for  extended  periods  of  time.  For  example,  a  typical  test 
may  require  Mach  number  to  be  held  +.01,  and  angle  of  attack  and  angle 
of  sideslip  to  be  held  within  -.25°,  during  a  level  turn  for  at  least 
one  minute.  Such  a  task  is  extremely  difficult  with  conventional 
displays,  requiring  significant  repeats  to  obtain  good  quality  data, 
and  subsequent  increases  in  expensive  flight  time.  With  a  special 
purpose  integrated  display,  however,  the  task  was  performed  with  rela¬ 
tive  ease  and  required  precision.  Nevertheless,  design  of  the  display 
was  a  time  consuming  "cut  and  try  on  the  simulator"  process.  Better 
information  about  pilot  dynamics  and  workload  would  greatly  facilitate 
the  design  process  for  this  application. 

In  conclusion  then,  we  have  seen  that  there  is  a  need  for  flight 
research  in  pilot  dynamics  and  workload  measurement.  Also,  pilot  dy¬ 
namics  and  workload  measurements  can  contribute  to  more  effective  human  factors 
flight  research  as  well  as  to  more  efficient  flight  testing  in  general. 
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PITCH  PIO  RATING  RESULTS 
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PIO  RATING  PIO  RATING 


HiMAT  OPERATIONAL  CONCEPT 
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AKEBED  SKID  LANDING  RECOVERY 
EDWARDS  AFB 


SECTION  II 


TAILORING  FLIGHT  TESTS  TO  IDENTIFY  PILOT  WORKLOAD 


Abstract  For  EAFB  Workload  Workshop, 


19-21  January  1981 


MEASURING  AIRCREW  WORKLOAD: 
PROBLEMS,  PROGRESS,  AND  PROMISES 

Henry  R.  Jex 
Systems  Technology,  Inc. 
Hawthorne,  California  90250 


An  overview  of  the  problems  of  defining,  quantifying,  and  measuring  mental 
workload  during  aircrew  tasks  is  given  based  on  our  work  in  the  areas  of  air¬ 
craft  handling  qualities,  pilot  model  measurement  and  prediction,  multi- 
display  scanning  and  psychophysiological  correlates  of  workload.  The  contin¬ 
ued  promise  and  problems  with  psychopysiological  measures  is  assessed  and  the 
importance  of  some  new  multidimensional  workload  rating  techniques  is  empha¬ 
sized.  The  lack  of  unifying  theoretical  approach  is  identified  as  the  main 
impediment  to  progress,  and  an  approach  is  suggested,  that  can  handle  both 
continuous  and  discrete  task  loads.  A  review  is  given  of  some  new  workload 
measurement  concepts  such  as  Non-Invasive  Pilot  Identification  Program,  the 
"imbedded  surrogate"  auxiliary  task  method,  and  the  measurement  of  workload 
margin  via  the  Cross-Coupled-Instability  Task  (CCIT). 
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Achieved  Performance 


CRITERIA  FOR  WORKLOAD  MEASURES 


1.  RELEVANT: 

TO  PROBLEM  &  SOLUTION 

2.  SENSITIVE: 

Monotonic  w.r.t  Subjective  WL,  High  "test- 
power"  WRT  WL  VARIABLES,  INSENSITIVE  TO  OTHER 
VARIABLES 


3.  CONCORDANT: 

Universal  effects  in  target  population 

A.  RELIABLE: 

TEST-RETEST 

"DIFFERENTIAL  STABILITY"  WRT  PRACTICE 
VALIDATED;  WITH  NORMS 


5.  CONVENIENT: 

PORTABLE 
EASY  TO  LEARN 
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CROSS  ADAPTIVE  AAEASURE  OF  EXCESS  CONTROL  CAPACITY  FOR 
SEVERAL  EXAMPLES  OF  PRIMARY  CONTROLLED  ELEMENTS 
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X*(rad/sec) 


TYPICAL  APPLICATION  OF  ADAPTIVE-WORKLOAD  TESTING 


Portion 
Due  to: 


Error: 
cr  i 


Workload: 


Operator 

Lead 

Req't 


None 


Some 


Much 


EVALUATION  OF  A  PILOT  WORKLOAD  ASSESSMENT  DEVICE 
TO  TEST  ALTERNATE  DISPLAY  FORMATS  AND 
CONTROL  HANDLING  QUALITIES 

by 

Samuel  G.  Schiflett,  Ph.D. 

Naval  Air  Test  Center 
Patuxent  River,  Maryland 

Paul  M.  Linton 

Naval  Air  Development  Center 
Warminster,  Pennsylvania 

Ronald  J.  Spicuzza 
Systems  Research  Laboratories 
Dayton ,  Ohio 


SUMMARY 

This  in-flight  research  project  evaluated  the  utility  of  a  Workload 
Assessment  Device  (WAD)  to  measure  pilot  workload  for  approach  and  landing 
tasks  under  simulated  instrument  meteorological  conditions,  alternate  HUD 
formats  and  control  stability  variations.  The  flight  tests  were  conducted 
in  an  NT-33A  research  aircraft,  extensively  modified  for  the  U.  S.  Air  Force 
and  U.  S.  Navy  by  the  Display  Evaluation  Flight  Test  program.  The  hardware, 
software,  and  test  procedures  associated  with  the  WAD  functioned  efficiently 
with  only  minor  discrepancies  and  minimum  pilot  distraction.  The  project 
established  the  feasibility  of  using  an  item-recognition  task  as  a  measure 
of  sensory-response  loading  and  reserve  information  processing  capacity 
while  flying  precision  approaches.  In  a  descriptive  statistical  treatment 
of  the  data,  the  results  indicate  an  appreciable  increase  in  reaction  time 
and  errors  with  degraded  handling  qualities  as  compared  to  ground  baseline 
measures  and  good  handling  qualities.  The  preliminary  findings  also  reveal 
consistent  trends  toward  the  availability  of  more  mental  reserve  capacity 
when  flying  predominantly  pictorial/symbolic  HUD  configurations  as  compared 
to  conventional  HUD  formats  with  scales  and  alphanumerics .  It  is 
recommended  that  further  evaluations  be  conducted  to  establish  the  efficacy 
of  utilizing  the  WAD  to  measure  mental  workload  in  a  wide  variety  of 
aircrew  tasks. 


INTRODUCTION 


BACKGROUND 


New  developments  in  cockpit  display  designs  and  integrated  weapons  system  avionics 
have  significantly  altered  the  role  of  the  pilot  from  that  of  a  skilled,  manual  control 
operator  to  an  executive  manager  of  an  integrated  weapons  system.  Emphasis  on  psycho¬ 
motor  control  has  been  augmented  by  an  interest  in  more  cognitive  skills  represented  by 
such  functions  as  short-term  memory,  information  processing,  and  decision  making.  Few 
measurement  techniques  exist  which  are  able  to  provide  an  objective,  reliable,  and  valid 
estimate  of  the  subtle  differences  in  workload  introduced  by  these  new  systems.  To  date, 
methodology  for  objectively  quantifying  workload  has  not  been  effectively  applied  to  the 
flight  test  and  evaluation  of  aircrew  systems  (references  1  and  2) . 

This  project  introduced  a  novel  approach  to  the  traditional  manner  of  measuring  pilot 
workload.  Aircrew  workloads  are  typically  measured  by  subjective  assessment  rating  scales 
which  are  based  on  pilot  opinions  that  relate  operational  task  demands  to  system  response 
characteristics,  e.g. ,  Cooper-Harper  Handling  Qualities  Rating  Scale.  The  new  approach 
applied  in  this  project  is  an  item-recognition  task  first  identified  by  Sternberg 
(reference  3)  and  modified  by  the  U.  S.  Air  Force  (reference  4)  to  measure  the  reserve 
capacity  of  the  pilot.  The  approach  assumes  that  an  upper  bound  exists  on  the  ability  of 
the  pilot  to  gather  and  process  information.  As  the  pilot's  workload  increases  on  the 
primary  task,  i.e.,  flying  the  aircraft,  reserve  capacity  for  processing  secondary 
information  decreases  until  a  point  of  overload  is  reached  by  the  pilot.  At  this  point, 
the  information  processing  demands  of  the  task  exceed  the  pilot's  total  workload  capacity 
and  is  manifested  by  degradation  in  performance  (i.e.,  increase  in  errors  and  response 
times)  on  the  secondary  item- recognition  task. 

The  theoretical  formulation  of  the  item- recognition  task,  as  proposed  by  Sternberg 
(figure  1),  has  several  attractive  features  which  make  it  ideally  suited  for  evaluating 
the  source  of  increase  in  task-loading  in  aircraft  test  environments.  The  theory  assumes 
a  least-squares,  linear  regression  fit  of  the  data  where  the  intercept  represents  the 
iriput/output  component  and  the  slope  depicts  the  mental  information  processing  component 
of  the  item-recognition  task.  If,  for  example,  the  sensory-response  mode  (i.e.,  input/ 
output) ,  is  response  overloaded  the  theoretical  expectation  is  a  change  in  the  y-intercept 
of  the  regression  line  with  no  change  in  slope.  Conversely,  if  the  source  of  task-loading 
was  one  which  affected  the  pilot's  mental  information  processing  capabilities  (e.g.,  short¬ 
term  memory  overload) ,  the  expectation  is  a  change  in  the  slope  of  the  curve  without  a 
corresponding  change  in  the  intercept  value.  Either  result  would  be  a  decrease  in  the 
pilot's  reserve  capacity  for  processing  information. 
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The  use  of  the  item-recognition  task  to  assess  primary  task  workload  is  not  a  new 
concept  in  aircrew  flight  simulation  studies  (references  5  and  6) .  However,  the 
uniqueness  of  its  application  in  this  project  is  that  a  Workload  Assessment  Device  (WAD) 
that  generates  and  controls  the  secondary  item-recognition  task  was  designed,  fabricated, 
and  installed  in  a  NT-33A  research  aircraft  to  measure  and  analyze  the  pilot's  reserve 
workload  capacity  for  the  Display  Evaluation  Flight  Test  (DEFT)  program  as  reported  in 
reference  7. 

PURPOSE 

The  purpose  of  this  project  was  to  evaluate  the  utility  of  the  WAD  to  measure  pilot 
workload  for  approach  and  landing  tasks  under  simulated  Instrument  Meteorological 
Conditions  (IMC's)  for  alternate  HUD  formats  and  aircraft  control  stability  variations. 

DESCRIPTION  OF  AIRCRAFT/EQUIPMENT 

The  NT-33A  variable  stability  aircraft  is  an  extensively-modified,  T-33  jet  trainer. 
The  elevator,  aileron,  and  rudder  controls  in  the  front  cockpit  were  disconnected  from 
their  respective  control  surfaces  and  connected  to  separate  servo-mechanisms  that 
comprise  an  "artificial  feel"  system.  In  addition,  the  elevator,  aileron,  and-  rudder 
control  surfaces  were  connected  to  individual  servos  which  were  driven  by  a  number  of 
different  electrical  inputs.  This  arrangement,  through  a  response-feedback  system, 
allowed  the  normal  T-33  stability  derivatives  to  be  augmented  to  the  extent  that  the 
handling  qualities  of  the  hypothetical  research  configurations  could  be  simulated.  A 
more  comprehensive  description  of  the  NT-33A  can  be  found  in  reference  8. 

The  DEFT  program  also  provided  a  fully  software-programmable  display  system  to 
complement  the  variable  stability  features  of  the  host-modified  NT-33  Aircraft.  Relative 
to  the  aircraft  configuration,  the  DEFT  system  provided  the  capability  of  changing 
display  formats  and  changing  the  algorithms  and  dynamics  of  the  display  driving  signals. 
The  display  system  consisted  of  a  HUD,  two  digital  computers,  a  magnetic  tape  system,  INS 
sensors  to  augment  the  existing  aircraft  sensors,  and  a  display  repeater  and  mode  control 
unit  for  the  aft  cockpit. 

The  software  programs  provided  an  in-flight  choice  of  two  uniquely  different  display 
configurations  for  use  in  the  approach  and  landing  phases  of  flight.  These  displays  were 
of  a  conventional  HUD  format  (figure  2)  and  the  predominantly  symbolic  Klopfstein  format 
(figure  3).  As  depicted  in  the  figures,  the  conventional  display  used  a  HUD  format  with 
a  flight  path  ladder,  scales,  and  alphanumeric  readouts  of  various  flight  parameters. 

The  Klopfstein- display,  however,  is  predominantly  symbolic/pictorial  depicting  the 
horizon,  and  artifical  runway  overlaying  the  actual  runway,  and  other  flight  guidance 
symbols. 

METHOD 


After  several  practice  sessions  and  prior  to  the  start  of  the  evaluation  flights,  a 
baseline  measurement  was  obtained  on  the  item-recognition  task.  Each  pilot  was  given  the 
item-recognition  task  for  each  memory  set  size  while  sitting  in  the  cockpit  of  the 
aircraft  stationed  on  the  ground.  The  task  required  the  pilot  to  memorize  sets  of  one, 
two,  or  four  letters,  i.e..  A,  RJ,  ZPNW.  The  pilot  was  then  instructed,  prior  to  testing 
with  each  memory  set,  which  set  of  letters  would  be  presented  for  memory  recall.  The 
prememorized  letters  (positive)  or  other  letters  (negative)  were  presented  on  the  HUD 
one  at  a  time  every  7  sec.  The  positive  and  negative  letters  were  presented  individually 
with  a  .5-probability  of  occurrence.  Each  letter  appeared  on  the  HUD  one  at  a  time  until 
the  pilot  responded  or  5  secs,  elapsed.  The  pilot  responded  to  a  letter  presentation  by 
pressing  one  of  two  designated  buttons  on  the  control  stick.  One  button  indicated  that 
the  letter  was  a  member  of  the  prememorized  set  (positive)  and  the  other  indicating  it 
was  not  a  member  of  the  prememorized  set  (negative) .  Positive  letters  never  appeared  as 
negative  letters  and  the  same  positive  letter  sets  were  used  throughout  the  test.  A 
total  of  30  letters,  15  positive  and  15  negative,  was  presented  for  each  memory  set  for 
the  baseline  conditions. 

The  same  procedures  were  used  in  flight  as  during  the  baseline  test  conditions  with 
the  exception  that  the  pilot  was  flying  the  aircraft  while  performing  the  secondary  task. 
An  additional  experimental  control  allowed  one  approach  per  handling  quality /display 
format  combination  to  be  flown  without  any  letter  presentations  to  evaluate  the  impact  of 
the  secondary  task  on  the  primary  task  of  flying  the  aircraft. 

The  reaction  times  and  response  errors  were  collected  and  analyzed  by  the  WAD 
controller  and  ground-based  analysis  system.  After  each  response,  the  reaction  time  was 
measured  from  the  onset  of  a  letter  to  the  physical  response  of  pressing  the  correct 
button.  The  reaction  times  for  both  the  positive  and  negative  letters  were  stored  on 
cassette  tapes.  The  reaction  times  for  the  correct  responses  were  then  averaged  and 
plotted  as  a  function  of  the  memory  set  sizes.  The  response  errors  were  coded,  tabulated, 
and  categorized  by  type  of  error  and  frequency  of  occurrence.  A  response  was  considered 
an  error  if  the  pilot  pressed  the  wrong  key  (reversal  error) ,  responded  correctly  but 
after  1,500  msec  (out-of-bound  error),  or  did  not  respond  before  5  sec  (time-out  error). 

The  basic  flight  scenario  for  each  approach  and  touch-and-go  was  as  follows.  The 
Evaluation  Pilot  (EP)  was  given  control  of  the  aircraft  by  the  Safety  Pilot  (SP)  with  the 
desired  display-aircraft  handling  quality  combination.  The  EP  then  flew  on  instruments 
while  'using  an  orange  filter  over  the  windscreen  and  a  blue  visor  attached  to  the  helmet 
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to  simulate  IMC.1  After  intercepting  the  glide  slope,  the  EP  descended  to  1,800  meter 
MSL  to  intercept  the  localizer  at  8  nmi.  At  this  point,  the  SP  turned  on  the  digital 
recorder  and  the  WAD  controller  which  were  used  to  record  the  primary  flight  measures 
and  the  secondary  task  measures,  respectively.  The  EP  proceeded  to  fly  the  glide  slope 
and  the  localizer  to  perform  the  approach.  The  outer  marker  was  at  approximately  4 
nmiles.  At  200  meter  AGL  and  approximately  1/2  nmi  from  the  runway  threshold,  the  EP 
"broke  out"  (i.e.,  he  lifted  the  blue  visor)  and  flew  visually  for  the  remainder  of  the 
low  approach  (7  meters  AGL).  If  conditions  permitted  (fuel  state,  crosswind,  etc.),  the 
EP  then  performed  the  touch-and-go  landing,  minimizing  the  sink  rate  on  touchdown  to  less 
than  1  meter/sec.  The  touchdown  point  was  a  170-meter  zone,  500  meters  from  runway 
threshold.  After  liftoff  and  at  approximately  70-meter  AGL,  the  SP  turned  off  the  WAD 
controller  and  the  digital  recorder.  After  four  approaches,  the  SP  assumed  control  of 
the  aircraft,  then  changed  the  pitch  handling  quality  to  the  next  desired  setting  and 
again  released  control  of  the  aircraft  to  the  EP. 

After  each  block  of  four  approaches  was  completed  under  the  same  pitch  handling 
quality,  the  EP  and  SP  rated  the  approach  and  flare/landing  segments  of  the  flight 
profile  using  the  Cooper-Harper  pilot  rating  scale.  Additional  commentary  data  were 
gathered  from  the  EP  and  SP  throughout  the  flight  tests  by  use  of  an  audio  tape  recorder, 
e.g.,  comments  on  degree  of  air  turbulence. 

The  WAD  consists  of  two  basic  units:  the  airborne  controller  and  the  ground-based 
analysis  center.  The  controller  is  configured  for  installation  in  the  front  avionics  bay 
of  the  NT-33A  research  aircraft.  The  unit  provides  the  electronics,  power  supply, 
software,  interfaces  to  the  HUD  and  the  aircraft  intercom,  rear  cockpit  initialization 
switches,  control  stick  response  switches,  and  data  recording  system  necessary  to  perform 
a  complete  series  of  item- recognition  experiments.  In  addition,  the  controller  can 
operate  as  a  stand  alone  laboratory  system  capable  of  performing  the  same  tasks  as  when 
airborne.  The  ground-based  data  analysis  center  is  used  to  initialize  several  software 
options  of  the  controller  and  to  reduce  and  analyze  response  time  data.  A  description  of 
the  functional  capabilities  of  the  hardware  and  software  is  discussed  in  appendix  A.  A 
detailed  description  of  the  complete  WAD  system  is  contained  in  reference  9. 

SCOPE 

Each  pilot  flew  two  evaluation  flights  using  the  conventional  HUD  format  and  two  with 
the  Klopfstein  format.  During  each  evaluation  flight,  a  pilot  performed  eight  approaches 
terminating  in  either  a  low  approach  or  touch-and-go  landing  for  a  total  of  32  approaches 
per  pilot.  One-half  of  the  approaches  for  each  flight  were  made  using  "good"  handling 
qualities,  the  other  half  were  made  using  either  "fair"  or  "poor"  handling  qualities. 

The  handling  qualities  were  manipulated  by  changing  the  pitch  response  (150  msec  or  200 
msec  time  delay)  of  the  aircraft  after  every  four  approaches.  The  response  of  the  roll 
and  yaw  axes  was  held  constant  throughout  the  tests. 


RESULTS  AND  DISCUSSION 

GENERAL 

The  test  and  evaluation  paradigm  used  in  this  project  was  a  repeated  measures  design 
in  which  type  of  display  format  (conventional  versus  Klopfstein) ,  flight  handling  quality 
(good  versus  poor),  and  secondary .task  difficulty  (memory  set  sizes,  0,  1,  2,  and  4)  were 
fractionally  combined  to  form  16  different  conditions.  It  was  planned  that  the  two  EP's 
would  be  exposed  to  each  of ‘  the  16  conditions  twice.  However,  each  EP  was  able  to 
complete  all  combinations  of  the  test  conditions  only  once.  Out  of  a  total  of  eight 
1.5-hr  evaluation  flights,  a  complete  set  of  secondary  task  data  was  analyzed  for  four 
flights  only. 

The  results  showed  that  the  general  procedures  established  for  the  conduct  of  the 
evaluation  flight  tests  of  the  WAD  were  acceptable  to  the  pilots.  The  in-flight  test 
procedures  provided  the  EP's  and  SP's  with  reliable  guidelines  for  efficient  and  safe 
crew  coordination  during  successful  approaches  and  during  incidents  of  all  equipment 
malfunctions.  Pilot  comments  aided  in  the  investigation  of  the  most  salient 
characteristics  of  the  item-recognition  task  including  the  selection,  location,  and 
timing  of  the  letters  as  presented  on  the  HUD.  A  thorough  testing  of  the  WAD  procedures 
during  the  project  resulted  in  only  minor  software  changes  and  hardware  replacements  and 
clearly  established  the  feasibility  of  using  the  item-recognition  task  for  in-flight  tests 

PRIMARY  FLIGHT  MEASURES 

The  primary  flight  measurement  data  taken  from  the  digital  recorder  were  divided  into 
two  defined  categories  of  approach  and  flare/landing.  Because  of  the  length  and  com¬ 
plexity  of  the  analyses  of  the  primary  flight  measurement  data,  the  results  were 
published  under  separate  cover  in  reference  10. 

The  summary  results  of  these  analyses  indicate  that  the  primary  flight  performance 
parameters  and  Cooper-Harper  ratings  showed  a  general  inconsistency  between  displays 
and  handling  qualities  during  the  approach  and  flare/landing  phases  of  the  flight  task. 

1  Overlaying  the  two  complementing  colors  produced  a  perceptual  environment  similar 
to  night  IMC  when  the  pilot  attempted  to  view  the  external  world. 


224' 


Lack  of  systematic  differences  in  the  primary  flight  measures  and  Cooper-Harper  ratings 
suggests  that  pilot  performance  remained  the  same  for  all  conditions.  That  is,  no  sig¬ 
nificant  differences  were  found  in  the  primary  flight  measures  between  display  formats, 
handling  qualities,  or  memory  set  sizes.  These  findings  indicate  that  the  pilots  com¬ 
pensated  for  the  increased  task  difficulty  by  maintaining  primary  flight  performance  at 
an  acceptable  level.  However,  this  pilot  compensation  was  not  without  cost.  A  loss  of 
information  processing  reserve  capacity  can  be  clearly  shown  from  the  results  of  the 
secondary  task  measures. 

SECONDARY  TASK  MEASURES 

Secondary  task  measures  consisted  of  reaction  times  in  which  slopes  and  intercepts 
were  calculated  after  solving  linear  regression  equations  for  each  set  of  data. 
Secondary  task  errors  for  the  item  recognition  task  were  calculated  for  all  incorrect 
responses,  late  responses,  and  no  responses. 

REGRESSION  EQUATIONS  (REACTION  TIMES) 


The  reaction  times  associated  with  each  correct  response  were  averaged  for  the 
complete  flight  profile  for  each  m-set  size  (letters  1,  2,  or  4),  handling  quality  (good 
or  poor) ,  and  display  format  (conventional  or  Klopfstein) .  Linear  regression  equations 
were  then  calculated  to  indicate  the  slope  and  intercept  of  the  plotted  data  as  shown  in 
figures  4  and  5.  The  data  reveal  that  both  the  intercept  and  slope  of  the  curves  for 
each  pilot  increased  from  baseline  conditions  when  the  handling  qualities  were  degraded. 
The  results  indicate  that  the  WAD  is  sensitive  to  the  increased  sensory /response  and 
mental  processing  requirements  imposed  by  the  addition  of  a  secondary  task  and  to  the  level 
of  difficulty  of  that  task.  For  example,  the  largest  intercept  and  slope  changes 
occurred  between  each  subject's  relative  baseline  and  poor  handling  quality  condition. 

A. closer  examination  of  the  data  reveals  that  the  differences  in  the  magnitude  of 
change  in  the  slopes  were  consistently  larger  for  the  conventional  HUD  format  than  the 
pictorial  Klopfstein  HUD  format  under  either  good  or  poor  handling  qualities.  This  trend, 
relative  to  each  subject's  shift  in  slope  magnitude,  suggests  that  more  mental  reserve 
capacity  was  available  to  process  information  while  flying  the  Klopfstein  display  than 
the  conventional  HUD  format  and  while  good  handling  qualities  independent  of  the  type  of 
display  format  used. 

Reviewing  the  resulting  changes  in  intercepts  revealed  a  similar  trend  with  regard  to 
the  handling  quality  parameter.  The  average  increase  in  the  magnitude  of  change  in  inter¬ 
cept  was  less  for  conditions  of  good  handling  qualities  than  for  poor  handling  qualities. 
However,  with  regard  to  the  display  variable,  the  trend  was  reversed  from  that  observed 
for  the  changes  in  slope;  i.e.,  the  average  intercept  value  changed  less  for  the  con¬ 
ventional  format  than  for  the  Klopfstein.  Assuming  the  observed  trends  would  persist  in 
a  larger  data  sample,  the  results  indicated  that  degrading  the  handling  qualities  had  a 
consistent  effect  on  the  input/output  stages  of  the  item-recognition  task,  whereas  the 
effect  of  the  display  format  variable  on  the  input/output  stages  of  the  task  was  subject 
to  inconsistent  individual  differences.  The  lack  of  consistent  trends  in  the  changes  in 
intercept  relative  to  the  display  variable  may  be  due  to:  (1)  individual  differences  in 
establishing  a  time-error  tradeoff, 2  (2)  locations  of  the  letter  in  relationship  to 
differences  in  eye  scan  patterns,  and/or  (3)  different  strategies  of  memory  recall. 

These  results  suggest  that  degrading  handling  qualities  had  a  consistent  and 
predominant  effect  of  reducing  the  pilot's  reserve  capacity  for  all  three  stages  of  the 
information-processing,  secondary  task.  Changing  the  display  formats  appeared  to  yield 
similar  results  but  are  subject  to  the  influences  of  individual  differences  with  regard 
to  the  mental  component  of  the  information  processing  task. 

The  reader  is  reminded  that  these  data  reveal  only  trends  and  were  gathered  from  a 
sample  of  two  pilots.  Additional  flight  data  are  required  with  a  larger  pilot  sample 
and  more  replications  of  test  conditions  before  definitive  conclusions  can  be  made  con¬ 
cerning  the  reliability  of  the  results  of  these  measures.  A  further  discussion  of  the 
reliability  of  the  item- recognition  task  that  questions  the  day-by-day  stability  of  the 
slope  and  intercept  is  found  in  reference  11. 

PERCENT  ERRORS 

The  WAD  provided  an  accumulative  record  of  the  number  of  errors,  sequence  of 
occurrence,  type  of  error,  and  reaction  time  associated  with  each  error  for  both  positive 
and  negative  letters.  The  combined  percent  of  secondary  task  errors  for  both  pilots  is 
shown  in  figure  6.  The  error  data  show  that  as  the  difficulty  of  the  secondary  task  was 
increased,  i.e.,  as  the  m-set  size  increased,  a  corresponding  decrease  in  response 
accuracy  was  observed  which  supports  the  expectation  of  increased  error  rate  under  con¬ 
ditions  of  task  overloading. 

The  increases  found  in  secondary  task  response  errors  under  conditions  of  poor  han¬ 
dling  qualities  for  both  display  formats  are  consistent  with  the  results  of  the  slope  and 
intercept  reaction  time  data  with  regard  to  the  influence  of  degraded  handling  qualities. 


2  The  EP's  were  only  instructed  to  respond  as  quickly  and  accurately  as  possible  to 
the  secondary  task  while  flying  a  precision  approach  and  landing. 
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That  is,  under  test  conditions  producing  a  reduction  in  reserve  capacity,  a  corresponding 
increase  in  response  errors  occurred. 

In  contrast,  the  reaction  time  data  indicated  that  the  type  of  display  format  differ¬ 
entially  influenced  both  the  input/output  and  mental  stages  of  the  information  processing 
tasks,  whereas  response  error  data  showed  a  consistently  higher  degree  of  response 
accuracy  under  conditions  of  the  pictorial  Klopfstein  display  format. 

To  further  explore  these  results,  the  total  percent  errors  were  classified  into  type 
of  error  for  each  handling  quality  and  display  format.  The  secondary  task  errors  reveal 
that  the  total  percent  errors  were  evenly  distributed  between  incorrect  responses 
(reversals) ,  late  responses  (out-of-bounds) ,  and  no  responses  (time-outs) .  However  when 
the  total  percent  errors  are  differentiated  between  display  type  and  handling  quality,  it 
clearly  shows  that  three  times  as  many  reversal  errors  were  committed  by  the  EP's  flying 
the  conventional  HUD  than  the  Klopfstein  display  format.  Degrading  the  handling  qualities 
increased  the  percentage  of  time-out  errors  for  the  EP's  flying  with  the  conventional 
display  and  increased  the  out-of-bounds  under  the  Klopfstein  display  format.  Since  it 
was  assumed  that  a  time-out  error  would  reflect  a  greater  decrement  in  reserve  capacity 
than  an  out-of-bounds  error,  these  results  would  imply  that  the  EP's  had  less  reserve 
capacity  while  flying  under  the  conventional  HUD  and  degraded  handling  qualities  than 
the  Klopfstein  display  format. 

In  summary,  the  percent  of  secondary  task  errors  increased  whenever  the  memory  set 
size  increased,  the  handling  qualities  were  degraded,  and  the  task  was  performed  in 
flight  under  the  conventional  display  format  conditions.  Poor  handling  qualities 
primarily  induced  errors  of  delay  or  no  response  while  the  type  of  display  affected  mainly 
the  accuracy  (correctness)  of  response. 


CONCLUSIONS 

The  hardware,  software,  and  test  procedures  associated  with  the  Workload  Assessment 
Device  (WAD)  functioned  efficiently  with  only  minor  discrepancies  and  minimum  pilot 
distraction. 

The  project  established  the  feasibility  and  sensitivity  of  using  a  secondary  item- 
recognition  task  as  a  measure  of  sensory/response  loading  and  reserve  information  pro¬ 
cessing  capacity  while  flying  precision  instrument  meteorological  conditions  approaches. 

The  pilots  showed  an  appreciable  increase  in  reaction  time  and  percentage  of  errors 
on  the  secondary  task  flown  under  poor  handling  qualities  as  compared  to  good  handling 
qualities  and  ground  baseline  conditions. 

The  WAD  revealed  that  the  pilots  had  less  secondary  task  errors,  more  mental  reserve 
capacity,  but  longer  reaction  times  attributed  to  sensory /response  delays  while  flying 
with  pictorial/symbolic  HUD  configurations  (Klopfstein)  than  conventional  HUD  formats. 
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REACTION  TIME 


SLOPE  =  INCREASE  IN  PROCESSING  WITH  EACH 


Figure  1  Theoretical  Components  of  the  Item 

Recognition  Task  Proposed  by  Sternberg 
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MEAN  REACTION  TIME  (MSEC) 


SECONDARY  TASK  ERRORS  (%) 


POOR  HANDLING  QUALITY 


GOOD  HANDLING  QUALITY 


Figure  6  Mean  Percent  Secondary  Task  Error  for 
Memory  Set  Size  (Number  of  Letters)  by 
Display  Format  and  Handling  Quality 
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WORKLOAD  ASSESSMENT  DEVICE 
SYSTEM  DESCRIPTION 


TASK 


The  Workload  Assessment  Device  (WAD)  presents  to  the  subject  an  item  recognition  task 
that  requires  him/her  to  respond  to  aural  or  visual  stimuli  that  are  composed  of  alpha¬ 
betic  characters  or  symbols.  After  a  stimulus  has  been  presented,  the  subject  responds 
by  pressing  one  of  two  switches  to  indicate  the  stimulus  is  (1)  part  of  a  pre-memorized 
set  of  letters,  or  (2)  is  not  part  of  the  set  (1) .  The  data  collected  are  the  reaction 
times,  in  milliseconds,  from  the  onset  of  a  stimulus  presentation  to  the  physical 
response  of  pressing  a  button. 

When  a  trial  is  completed,  the  subject  is  given  another  set  of  letters  to  memorize 
and  the  sequence  is  then  repeated.  Usually  four  trials  are  included  in  a  given  session 
with  the  memory  size  ranging  from  one  to  four  letters.  Data  analysis  consists  of  deter¬ 
mining  mean  reaction  time  to  a  number  of  presentations,  and  the  standard  deviation  of  the 
response  reaction  times. 

SYSTEM  CONFIGURATION 


Given  the  above  task,  its  complexity,  number  of  possible  deviations,  and  timing 
considerations,  a  microprocessor  was  chosen  for  the  main  controller.  Peripheral  devices 
are  manipulated  using  software  located  in  programmable  memory,  (RAM) .  All  data  collected 
are  temporarily  stored  in  memory  and  subsequently  recorded  on  a  digital  cassette  tape. 

When  the  mission  is  completed,  the  tape  is  retrieved  from  the  real-time  controller  and 
taken  to  a  ground-based  data  analysis  computer.  Prior  to  each  mission,  certain 
parameters  are  entered  into  the  ground-based  computer  and  written,  along  with  the  opera¬ 
ting  software,  to  the  cassette  tape.  These  parameters  are  used  by  the  main  controller  in 
order  to  present  variations  of  the  task  described. 

The  system  configuration  is  shown  in  Figure  A-l  and  consists  of  a  real-time  controller 
and  data  collection  device,  software,  and  a  ground-based  data  analysis  package. 

In  order  to  provide  for  portability,  a  chassis  was  constructed  to  fit  into  a  specific 
location  in  the  nose  of  the  NT-33A  aircraft.  The  chassis  is  small  enough  to  fit  into  a 
standard  off-the-shelf  enclosure  and  light  enough  to  be  hand  portable.  The  control  panel 
of  the  WAD  has  three  connectors  that  provide  all  interface  signals  required  for  operation. 
In  the  portable  mode,  these  connectors  provide  I/O  lines  that  can  be  interfaced  to  various 
display  devices  and  response  keys  such  as  used  in  many  simulators  and  laboratory 
environments . 

HARDWARE 

The  Workload  Assessment  Device  (WAD)  was  designed  around  the  IEEE  696.1/02  buss, 

(S-100) .  This  buss  configuration  was  chosen  for  the  size  of  the  printed  circuit  board, 
availability,  and  cost.  Most  S-100  devices  manufactured  today  are  reliable  and  well 
constructed,  and  there  exists  a  large  base  of  different  peripherals  to  choose  from. 

Since  this  system  is  experimental  and  cost  a  major  concern,  it  was  not  required  to  meet 
government/mi 1  specifications  for  reliability,  temperature,  and  vibration. 

The  WAD  mainframe  contains  5  slots  that  are  used  for  the  various  peripheral  interfaces, 
memory,  and  CPU.  A  single  board  computer  manufactured  by  Cromemco,  Inc.,  is  used  for  the 
main  controller.  It  contains  a  serial  I/O  port,  several  parallel  I/O  lines,  real  time 
clock.  Read  Only  Memory  (ROM) ,  Programmable  Memory  (RAM) ,  and  all  necessary  system  timing 
signals.  A  16K  RAM  board  is  used  for  program  and  data  storage.  The  next  buss  location 
contains  the  digital  cassette  interface  which  is  connected  to  the  NFE  Corporation  digital 
tape  recorder  located  on  the  front  panel  of  the  WAD.  The  fourth  slot  contains  a  16 
channel  analog  to  digital  (A/D)  converter  and  interface.  This  unit  is  connected  through 
cables  to  the  aircraft's  analog  computer  and  can  be  used  to  monitor  up  to  6  different 
control  surfaces  that  will  be  used  in  a  derivation  of  the  described  item  recognition  task. 
The  fifth  position  contains  the  speech  synthesizer  board.  This  board  contains  the  new 
National  Semiconductor  speech  processor  IC  along  with  its  ROM  vocabulary  ICs. 


SYSTEM  OPERATION 

When  power  is  applied  to  the  system,  a  boot  program  located  in  ROM  loads  a  file  from 
the  cassette  tape  recorder.  This  file  contains  a  program  that  controls  all  operations  of 
the  item  recognition  task.  After  loading,  the  program  gains  control  of  the  system  and 
waits  for  a  command  from  the  serial  port.  The  experimenter  has  several  options  at  this 
time.  Usually,  he  will  enter  a  command  for  the  system  to  load  a  specific  parameter  file 
from  the  tape.  This  file  contains  all  the  parameters  used  in  this  presentation  of  the 
task,  such  as  Inter-stimulus  Interval  (ISI) ,  Memory  Set  Size  (MSET) ,  use  of  visual  mode 
versus  auditory,  etc.  After  the  experimenter  enters  his  selection  the  subject  is 
presented  the  task.  During  the  task  presentation,  all  the  error  scores,  reaction  times, 
and  other  useful  data  are  stored  in  files  of  the  cassette  tape  containing  all  the  col¬ 
lected  reaction  times,  error  scores,  and  various  other  parameters.  The  program  then 
recycles  to  the  experimenter's  console  and  waits  for  another  command.  When  the  experi¬ 
mental  session  is  over,  the  cassette  tape  is  retrieved  for  preliminary  data  reduction 
and  display. 
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In  order  to  create  a  cassette  tape  containing  the  operating  program  and  stimulus 
parameters,  a  microcomputer  is  provided  that  contains  2  floppy  disk  drives,  mainframe, 
CRT  terminal,  printer,  digital  tape  recorder,  and  Disk  Operating  System  (DOS).  Located 
on  disks  are  several  user  programs  that  allow  the  experimenter/scientist  to  create 
parameter  files.  Also  located  on  the  disk  is  a  program  that  contains  the  operating 
software  for  the  WAD  along  with  a  linker.  When  the  experimenter  wants  to  create  a 
parameter  tape  he  links  together  the  previously  created  parameter  files  to  be  used  with 
the  task  and  the  WAD  operating  software.  This  newly  created  link  file  is  then  written 
out  to  the  cassette  tape. 

During  data  reduction  or  analysis,  the  cassette  tape  containing  the  r.ewly  recorded 
data  is  placed  in  the  tape  transport  and  a  loading  program  is  run.  This  program  creates 
files  on  the  disk  containing  all  the  experimental  data  collected.  The  experimenter  is 
then  able  to  display  the  data  on  the  CRT,  print  it  out  on  the  line  printer,  or  submit 
the  data  to  several  data  reduction  or  analysis  programs. 

SOFTWARE 


The  WAD  software  consists  of  several  programs  mentioned  above.  Most  of  the  hardware 
drivers  and  controlling  software  are  written  in  assembly  language,  but  some  of'  the  com¬ 
plex  data  handling  routines  are  written  using  Pascal.  All  of  the  data  analysis  software 
runs  under  the  CP/M  disk  operating  system  (DOS) .  This  DOS  was  chosen  because  many 
applications  software  packages  and  high  level  languages  are  designed  to  use  CP/M  for 
their  I/O  and  file  structures. 

EXPANDABILITY 

Since  the  main  controlling  software  for  the  WAD  is  located  at  the  beginning  of  each 
parameter  tape  and  the  source  is  on  the  floppy  disks,  it  is  very  easy  to  modify.  The 
experimenter  simply  makes  his  changes  using  the  text  editor  and  recompiles  the  program. 
When  he  makes  a  parameter  tape  the  new  controlling  software  will  be  included  provided 
the  file  name  was  not  changed.  This  provides  the  experimenter/scientist  with  a  very 
versatile  system  that  can  be  modified  for  custom  applications  and  has  the  capability  of 
adding  new  tasks.  Since  the  peripherals  provided  are  under  software  control,  any 
sequence  of  operation  can  be  programmed,  thus  allowing  many  different  tasks  to  be 
included  in  the  data  base. 

In  addition,  16  channels  of  analog  to  digital  (A/D)  converters  with  interface  are 
being  installed.  This  will  enhance  the  system  by  allowing  it  to  sample  up  to  six 
primary  control  surfaces  from  the  NT-33A,  or  any  other  system/simulator.  By  arranging 
the  data  under  software  control,  many  derivations  of  the  item-recognition  task  can  be 
constructed. 
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Abstract 

This  paper  describes  work  in  progress  on  the  use  of  visual  scanning 
behavior  as  an  indicator  of  pilot  workload.  The  study  is  investigating  the 
relationship  between  level  of  performance  on  a  constant  piloting  task  under 
simulated  IFR  conditions,  the  skill  of  the  pilot,  the  level  of  mental 
workload  induced  by  an  additional  verbal  task  imposed  on  the  basic  control 
task,  and  visual  scanning  behavior. 

The  results  indicate  an  increase  in  fixation  dwell  times,  especially 
on  the  primary  instrument  with  increased  mental  loading.  Skilled  subiects 
"stared"  less  under  increased  loading  than  did  novice  pilots.  Sequences  of 
instrument  fixations  were  also  examined.  The  percentage  occurrence  of  the 
subiect's  most  used  sequences  decreased  with  increased  task  difficulty  for 
novice  subiects  but  not  for  highly  skilled  subiects. 

Entropy  rate  (bits/sec)  of  the  sequence  of  fixations  was  also  used  to 
quantify  the  scan  pattern.  It  consistently  decreased  for  most  subiects  as 
the  four  loading  levels  used  increased.  An  exponential  equation  in  task 
difficulty  was  found  to  be  a-  good  predictor  of  entropy  rate.  When  solved 
for  task  difficulty,  the  equation  provided  an  estimate  of  the  level  of  task 
difficulty  perceived  by  A  subiect. 

Piloting  and  numbef  task  performance  measures  were  recorded  and  a 
combined  performance  measure  was  computed.  Skill  was  estimated 
independently  via  a  method  based  on  pilot  experience.  These  measures  were 
combined  with  entropy  rate  to  develop  a  model  relating  performance,  skill, 
and  mental  workload.  The  exponential  model  fit  the  data  well  enough  to 
suggest  that  this  approach  has  promise  in  the  evaluation  of  interactions 
among  these  variables. 

Introduction 

The  quantification  of  mental  workload  in  aircraft  pilots  has  been  of 
considerable  interest  for  some  time.  Perhaps  the  chief  reason  for 
measuring  workload  is  to  predict  conditions  under  which  task  performance 
will  decrement.  If  such  conditions  could  be  accurately  predicted,  then  the 
nature  and  temporal  sequence  of  flight  procedures  and  of  pilot/aircraft 
interfaces  might  be  arranged  so  as  to  minimize  the  chances  of  overload. 
Quantitative  analyses  of  workload  remain  elusive  however.  What  one  would 
like  is  a  clear  cause  and  effect  relationship  between  an  independent 
variation  in  imposed  workload  and  some  reliable  dependent  measure. 
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The  task  of  flying  an  aircraft  is  complex  however,  and  it  has  been 
difficult  to  clarify  the  functional  relationships  between  various 
parameters  in  piloting  tasks.  The  skill  a  particular  individual  brings  to 
the  piloting  task  and  the  nature  of  the  task  which  is  performed  can  both  be 
expected  to  influence  the  "difficulty"  of  the  task.  These  factors  may  be 
further  complicated  by  a  shift  in  the  pilot's  priorities;  (Some  tasks  may 
be  ignored  while  others  receive  full  attention). 


SKILL 


WORKLOAD 

Figure  1.  INTUITIVE  RELATIONSHIPS  BETWEEN 
PERFORMANCE,  SKILL,  &  WORKLOAD 


The  problems  which  such  inter-relationships  introduce  is  well 
illustrated  when  one  attempts  to  employ  task  performance  as  an  indicator  of 
workload.  All  pilots,  regardless  of  skill,  can  be  expected  to  exhibit  poor 
performance  #if  the  loading  level  is  excessive.  The  overload  situation  is 
relatively  easy  to  assess.  however.  using  subiective  techniques. 
Situations  which  involve  intermediate  to  high  levels  of  loading  would  seem 
to  be  the  ones  of  more  practical  concern;  i.e.,  one  is  concerned  with 
minimizing  the  chance  of  a  high  workload  approaching  an  overload  situation. 
Intuition  suggests  that  the  level  of  skill  of  the  pilot  may  influence  the 
performance  vs  workload  relationship  for  intermediate  or  marginal  loading 
levels.  A  pilot  of  high  skill  would  be  expected  to  maintain  "better" 


performance  than  a  novice  fiver  under  anv 


loading  condition  short  of 
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overload.  This  intuitive  concept  is  illustrated  Graphically  in  figure  1. 

The  research  described  here  uses  this  graphical  representation  of  the 
performance/skill/workload  relationships  in  order  to  pose  a  number  of 
testable  hypotheses.  It  will  be  suggested  shortly  that  instrument  scan  may 
be  an  indicator  of  workload  and/or  skill  in  certain  types  of  flight 
situations,  a  suggestion  supported  by  both  qualitative  and  quantitative 
results.  In  addition,  if  a  measure  of  workload  based  on  instrumnt  scan  is 
combined  with  independent  measures  of  pilot  skill  and  performance,  then  a 
model  of  the  hypothetical  relationships  in  figure  1  may  be  developed  and 
tested. 

Visual  Scanning  Behavior 

The  pilot  has  many  sources  of  information  input  but  the  most  important 
one  during  instrument  flight  is  probably  the  visual  pathway.  Under 
instrument  flight  conditions,  some  sensory  inputs  may  even  provide  false 
information  such  as  vertigo  which  results  from  conflicting  visual  and 
vestibular  information.  The  pilot  obtains  information  concerning  aircraft 
state  by  cross-checking  or  scanning  the  flight  instruments.  The  exact 
method  of  scanning  the  instrument  panel  varies  from  pilot  to  pilot  but 
there  are  some  basic  features  common  to  a  "good"  scan  pattern.  Indeed,  it 
was  the  early  study  by  Fitts  and  his  associates  on  instrument  transitions 
which  led  to  the  familiar  "T"  arrangement  of  the  maior  flight  instruments 
(Jones,  et . al. .  1946). 

A  fundamental  notion  n  the  present  work  is  that  a  repetitive  piloting 
task  will  invoke  a  regular  visual  scan  (spatial/temporal  pattern  of  eye 
movements)  during  instrument  flight.  If  this  notion  is  correct,  then  it 
may  be  postulated  that  external  factors  such  as  noise,  interruptions,  and 
fatigue  which  interfere  with  the  piloting  task  may  produce  measurable 
changes  in  the  scanning  behavior.  Such  a  measure  would  be  particularly 
attractive  for  quantifying  workload  since  it  would  be  both  non-invasive  and 
obiective. 

Experimental  Design 

A  series  of  experiments  is  being  carried  in  order  to  carefully  examine 
these  ideas.  The  basic  experiment  is  described  in  detail  elsewhere  (Tole. 
et  al.  1982)  and  only  the  salient  points  are  repeated  here.  The 
experiments  described  were  performed  at  the  NASA/Langley  Research  Center. 
Flight  Management  Branch,  in  Hampton.  Virginia,  making  use  of  their  flight 
simulator  and  oculometer  facilities  (Middleton,  et.al.,  1977). 

Three  factors  were  manipulated  in  the  experiments:  1)  a  piloting  task 
requiring  a  stereotyped  scan  path,  2)  a  verbally  presented  mental  loading 
task,  and  3)  a  workload  calibration  side  task. 

We  sought  a  representative  constant  piloting  maneuver  which  might  be 
realistically  expected  to  occur  for  periods  of  up  to  10  minutes  in  actual 
flight.  This  run  length  was  chosen  as  an  estimate  of  the  minimum  amount  of 
time  required  to  provide  a  sufficient  number  of  instrument  fixations  to 
satisfy  the  assumption  of  steady  state  conditions.  The  Instrument  Landing 
System  (ILS)  approach  is  often  chosen  as  the  piloting  task  in  studies  of 
workload  (Waller.  1976;  Krebs  and  Winaert.  1976;  Spady,  1977).  Hovever. 
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the  ILS  approach  represents  a  constantly  changing  task  difficulty  as 
touchdown  is  approached  (especially  due  to  increases  in  Glide  slope 
sensitivity  and  cost  of  error  for  course  deviation).  This  variation  in  the 
primary  task  loading  makes  it  difficult  to  accurately  control  the  amount  of 
mental  workload  on  the  pilot  as  an  independent  variable.  It  was  decided 
that  a  scenario  in  which  glide  slope  sensitivity  and  heading  were  held 
constant  would  allow  the  piloting  task  difficulty  to  remain  relatively 
constant  for  a  long  period,  but  nevertheless  be  more  or  less  realistic. 

A  desktop  general  aviation  instrument  flight  simulator  (Analog 
Training  Computers  ATC-510)  was  used  to  simulate  these  flight  manuevers. 
The  ATC-510  is  a  procedures  trainer  for  light,  single  engine,  fisted  Ditch 
prop,  fixed  gear.  IFR  equipped  aircraft.  The  simulator  was  equipped  with  a 
turbulence  level  control  which  was  set  to  the  first  level  above  calm 
conditions  in  order  to  force  some  pilot  vigilance  on  the  flight  task. 

Pilot  Iookooint  on  seven  instruments  (Attitude  Indicator  'ATT', 
Directional  Gyro  'DG1.  Altimeter  'ALT'.  Vertical  Speed  Indicator  'VSI', 
Airspeed  'AS1.  Turn  and  Bank  '*B'.  and  Glide  Slope/Localiser  'GSL')  was 
measured  using  a  Honeywell  oculometer  system  which  has  been  substantially 
modified  by  NASA  Langley  Research  Center  (Middleton,  et.al.,  1977).  This 
device  is  non-invasive  and  allows  the  user  to  determine  the  time  course  of 
eye  fixations  on  instruments  employed  by  the  pilot  and  the  dwell  time  of 
each  fixation  to  the  nearest  1/30  sec. 

The  mental  loading  task  was  chosen  so  as  not  to  directly  interfere 
with  the  visual  scanning  of  the  pilot  (i.e.  the  task  would  not  require  the 
pilot  to  look  away  from  the  instruments)  while  providing  constant  loading 
during  the  maneuver.  The  task  used  required  the  pilots  to  respond  to  a 
series  of  evenly  spaced  three-number  sequences  (Wittenborn.  1943)  presented 
to  them  audibly  by  means  of  a  speaker.  The  pilot  was  told  that  he  must 
respond  to  each  three-number  sequence  by  indicating  either  "plus"  or 
"minus"  according  to  the  algorithm  :  first  number  largest,  second  number 
smallest  =  "plus"  (e.g.  5-2-4).  last  number  largest,  first  number  smallest 

=  "plus"  (e.g.  1-2-3),  otherwise,  "minus"  (e  g.  9-5-1). 

The  mental  workload  experienced  by  the  pilot  is  inversely  proportional 
to  the  intervals  between  number  sequences.  This  relationship  is  given  by 
the  following  equation  which  is  arbitrarily  chosen: 

(1)  TD  =  1/interval  between-^  task 

where  TD  is  equal  to  imposed  task  difficulty.  The  four  loading  levels  used 
in  the  current  experiments  were  intervals  of  continuous  silence  (i.e. 
no-numbers  presented),  ten,  five,  and  two  seconds  which  have  corresponding 
task  difficulties  of  0.0,  0.1,  0.2,  and  0.5,  respectively. 

Numbers  were  generated  by  a  computer  controlled  speech  synthesizer. 
This  allowed  automated  scoring  of  task  accuracy,  calculation  of  response 
reaction  times,  and  the  possibility  of  temporal  correlations  of  visual  or 
other  responses  with  the  verbal  stimulus.  The  probabilities  of  occurence 
of  "  +  "  and  sequences  were  each  0.5.  The  pilot  was  instructed  to  give 

the  number  task  priority  equal  to  that  of  the  piloting  task  as  if  the 
verbal  questions  represented  a  constant  rate  of  radio  communication. 
Performance  was  recorded  by  having  the  pilot  press  a  3-position  rocker 
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switch  mounted  on  the  yoke  up  for  plus  and  down  for  minus. 


The  amount  of  mental  loading  imposed  on  the  pilot  by  the  number  task 
was  calibrated  using  a  side  task  <Ephrath,  1975).  Th6  runs  made  with  the 
side  task  were  not  used  in  the  scanning  analysis,  however,  due  to  the 
alteration  of  normal  scanning  caused  by  the  task.  The  results  (Tole, 
et.al.,  1982)  from  these  runs  confirmed  the  relative  difficulty  of  the 
various  number  intervals. 

A  microprocessor  development  system  (Burns,  et.al.  1980)  was  used  for 
both  stimulus  presentation  and  data  collection  and  analyses. 

Performance  Measures 

Several  variables  were  obtained  from  each  of  the  twotasks  in  order  to 
allow  the  computation  of  performance  scores.  The  scores  developed  ran 
between  0  percent  and  100  percent  with  100  percent  being  obtained  if  the 
pilot  never  deviated  from  the  intended  path  in  space  on  the  piloting  task, 
and  if  all  number  task  sequences  were  answered  correctly  for  the  mental 
loading  number  task.  The  scores  from  the  piloting  and  the  mental  loading 
tasks  were  then  combined  to  provide  a  performance  measure  to  be  used  in  the 
validation  of  proposed  perf ormance/skill/workload  model. 

The  scoring  measure  for  the  number  task  was  computed  as  given  below. 

(  TOT  -  WRO  -  MIS) 

(2)  4Pvv  - -  x  100% 

TOT 

where 

TP  =  mental  loading  number  task  performance 
TOT  =  total  number  of  stimuli  presented 
WRO  =  number  of  incorrect 'responses 
MIS  =  number  of  missed  responses 

This  score  was  100  peroent  if  the  pilot  answered  every  sequence  correctly 
and  zero  percent  if  a  pilot  either  answred  incorrectly  or  missed  all  of 
the  stimuli  presented.  Most  subiects  score  nearly  100%  on  this  task  if 
they  have  nothing  else  to  do  simultaneously. 

The  raw  data  available  for  scoring  performance  on  the  piloting  task 
were  the  errors  from  the  intended  track  for  the  glide  slope  and  localizer 
courses.  Discussions  with  several  highly  skilled  pilots  revealed  that 
accuracy  of  tracking  the  glide  slope  and  localizer  might  not  provide  a 
complete  performance  picture.  These  pilots  were  willing  to  trade  off 
"smoothness"  when  the  loading  task  became  more  difficult;  i.e.  the  pilot 
may  perform  the  piloting  task  to  the  same  level  of  accuracy.  as  far  as 
deviations  from  a'  designated  path  are  concerned,  on  two  different  runs  but 
produce  two  very  different  ride  qualities  for  these  runs.  One  possible 
measure  for  smoothness  could  be  the  frequency  of  oscillation  around  the 
intended  path.  The  higher  this  frequency  is,  the  less  "smooth"  the  ride 
becomes.  It  was  arbitrarily  assumed  that  a  smooth  ride  would  contain 
frequecies  mostly  less  than  0.1  Hz.  Under  this  assumption,  measurement  of 
the  spectral  component  of  the  aircraft  dynamics  above  0.1  Hz.  would 
indicate  any  decrement  in  the  ride  quality. 
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In  order  to  examine  this  measure,  the  power-spectral  density  (PSD)  of 
the  .  course  deviations  was  computed.  The  bandwidth  of  the  calculated  PSD 
was  2.5  Hr.  The  "power"  within  a  band  of  frequencies  may  be  determined  by 
integrating  the  PSD  over  that  band  (Schwartz.  1959).  We  chose  to  consider 
the  %  of  the  spectral  power  which  was  located  in  the  band  from  0.1  to  2.5 
Hz.  This  was  calculated  by  subtracting  the  power  contained  in  the  band 
from  0  to  0.1  Hz  (assuming  that  the  D.C.  component  was  first  removed)  from 
the  total  power  in  the  spectrum  and  multiplying  by  100%.  This  %  of  the  PSD 
was  computed  for  both  the  glide  slope  and  the  localizer  and  combined  wth 
the  two  RMS  measures  to  provide  four  candidate  variables  to  be  included  in 
a  performance  score  for  the  piloting  task. 

Since  the  pilots  were  instructed  to  give  equal  priority  to  the 
piloting  task  and  the  mental  loading  number  task,  both  were  included  in  the 
development  of  a  combined  performance  score.  While  a  weighting  of  0.5 
might  have  been  assigned  to  each  task,  it  was  decided  to  leave  the 
weighting  free  to  allow  the  model  fitting  procedure  to  determine  the 
relative  weights.  A  linear  relationship  between  all  of  the  terms  was 
assumed  and  the  form  of  the  equation  became: 

(3)  P  =  CONST  +  a(j^P)  +  b(RMS/GS)  +  c(RMS/LOC> 

+  d(%PWR/GS>  +  e(%PWR/LOC) 

where 

P  =  combined  performance  measure 
CONST  =  constant  term 
TP  =  mental  loading  number  task  performance 
RMS/GS  «=  RMS  error  from  glide  slope  track 
RMS/LOC  =  RMS  error  from  localizer  track 

%PWR/GS  =  percent  of  power  from  the  power-spectral  density  for 
the  glide  slope  greater  tan  0.1  Hertz 
'APWR/LOC  =  percent  of  power  from  the  power-spectral  density  for 
the  localizer  greater  than  0.1  Hertz 

Estimation  of  Pilot  Skill  levels 

In  order  to  assess  the  effects  of  skill  on  performance  and  mental 
workload,  an  independent  quantitative  measure  of  skill  was  needed.  A  model 
of  pilot  skill  based  on  experience  factors  was  used  for  this  purpose 
(Hollister,  et  al,  1973).  This  model  was  developed  in  order  to  predict  the 
current  level  of  skill  of  pilots  flying  light,  single  engine  aircraft. 

(4)  Skill  =  1.42  +  0.25(recency)  +  0.73(log(total  time)) 

-  0.030(vears  certified)  +  0.15(Iog(time  in  type)) 

-  0.0088(age)  +  e 

where 

Skill  =  score  reflecting  relative  piloting 
performance 

recency  =  number  of  flight  hours  in  past  30  days 
total  time  =  total  number  of  flight  hours 

time  in  type  =  total  number  of  hours  in  light  single  engine  aircraft 
years  certified  =  time  in  years  since  last  certificate 
orating 

age  =  subiects's  age  in  years 

e  =  residual  variance  not  explained  by  the  model 
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A  raw  skill  score  was  calculated  for  each  of  the  pilot  subiects  using 
the  model.  The  pilot  with  the  highest  resulting  skill  score  was  then  used 
to  normalize  all  of  the  scores  so  that  skill  levels  would  range  between  0% 
and  100%.  Eleven  subiects  ranging  in  skill  from  NASA  test  pilots  to 
non-pilots  participated  in  the  experiments.  The  relative  skill  scores  for 
the  subiects  are  given  in  Table  I. 

NASA  PILOT#  SKILL  SCORE 


3 

100% 

4 

85 

11 

77 

13 

53 

15 

39 

6 

37 

12 

33 

14 

32 

8 

22 

7 

15 

16 

13 

TABLE  I. 

Relative  Skill  Scores  of  Subiects  based  on  Equation  4 


Though  care  must  be  taken  when  applying  an  equation  such  as  this  in  a 
different  set  of  experimental  conditions,  the  overall  rank  ordering  of  the 
pilots  by  this  method  is  probably  accurate  as  it  generally  agreed  with 
subiective  rating  of  the  pilot's  skills  by  experienced  observers  at  the 
NASA/Langlev  Research  Center. 

Conduct  of  the  Experiments 

Each  session  consisted  of  four  10-minute  runs  with  a  5-minute  break 
between  each  run.  The  difficulty  of  the  mental  loading  task  would  start  at 
no  numbers  for  the  first  run  and  increase  to  2-sec  intervals  by  the  fourth 
run.  Some  subiects  participated  in  two  sessions,  one  without  and  one  with 
the  side  task.  Each  subiect  was  allowed  to  practice  all  three  tasks  until 
he  felt  comfortable  with  them. 


Preliminary  Results 


Instrument  dwell  time  histograms  and  the  frequency  of  usage  of 
different  sequences  of  instrument  fixations  were  both  affected  by  the 
loading  task.  Both  results  are  reported  in  detail  elsewhere  (Tole,  et.al., 
1982)  and  ,  only  the  maior  points  are  mentioned  here.  An  increase  in  dwell 
time  with  increase  in  mental  loading  was  observed  in  all  subiects.  This  is 
illustrated  in  figure  2.  Novice  subiects  generally  had  much  longer  dwell 
times  under  increased  load  than  did  skilled  pilots,  (Relative  skill  levels 
are  given  in  Table  I  above.)  -The  fixation  sequences  of  the  pilot's 
instrument  sans  were  analyzed,  and  the  percentage  occurrence  of  the  ten 
most  frequently  occurring  sequences  were  also  analyzed.  These  results 
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Figure  2.  DWELL  TIME  HISTOGRAMS  FOR  TWO  SKILLED  PILOTS  (#4  &  #11) 
AND  TOO  NOVICE  PILOTS  (#9  &  #10)  UNDER  VARIOUS.  LOADING 
CONDITIONS 

indicate  that:  1)  skilled  pilots  use  a  higher  percentage  of  their  ten  most 
frequently  occurring  sequences  than  do  novice  pilots  and  Z>  the  scan 
pattern  of  the  novice  subiects  were  affected  more  by  the  increase  in  mental 
loading  than  were  the  patterns  of  the  highly  skilled  pilots.  This  result 
is  shown  in  figure  3. 

A  more  general  method  of  quantifying  the  scan 

Traditionally,  much  of  the  quantitative  analysis  of  scanning  patterns 
has  employed  Markov  transition  probability  matrices  (Stark  and  Ellis.  1981; 
Krebs  and  Wingert.  1976).  Such  matrices  do  describe  the  predominant 
patterns  in  the  scan  via  the  relative  sizes  of  transition  probabilities  but 
it  is  either  extremely  unwieldy  or  impossible  to  compare  two  of  these 
matrices  for  different  experimental  conditions.  One  of  the  maior  goals  of 
this  research  is  the  identification  of  general  methods  for  the  study  of 
scanning  behavior.  To  be  most  useful  the  method  should  be  independent  of 
the  number  and  arranament  of  instruments.  The  nature  of 

eve-point-of-regard  data  (sequential  instrument  and  dwell  times)  obtained 
from  the  oculometer  suggests  several  methods  from  information  theory  which 
may  have  this  generality. 
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LOADING  INTERVALS  INTERVALS 
LOADING  TASK 

Figure  3.  PERCENT  USAGE  OF  LENGTH  4  SEQUENCES  UNDER  VARYING 
LOAD  (TYPICAL  SEQ  :  ATT  -  DG  -  ATT  -  ALT) 


The  piloting  task  in  the  current  experiment  is  such  that  the  pilot's 
scan  can  only  lie  on  one  of  the  7  specified  instruments  although  each 
fixation  may  he  of  arbitrary  duration.  The  time  history  of  fixations  has  a 
form  which  is  similar  to  that  of  a  communications  system  which  can  assume  7 
discrete  states  with  a  varying  duration  in  each  state.  The  orderliness  of 
such  a  system  is  related  to  the  probabilities  with  which  it  occupys  its 
different  states.  A  system  which  always  occupied  the  same  state  or  always 
made  the  same  transitions  between  states  would  thus  be  quite  orderly.  In 
the  case  of  instrument  scan,  these  situations  would  be  paralleled  by 
staring  and  by  a  stereotyped  scanpath  respectively. 

This  concept  of  system  order  may  be  stated  compactly  using  the 
mathematical  form  for  entropy  from  information  theory.  The  entropy  of  a 
sequence  is  defined  as  (Shannon  and  Weaver.  1949): 
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D 

(5)  H  =  p  log  p 

o  i=l  i  2  i 

where 

H  =  observed  average  entropy 
o 

p  =  probability  of  sequence  i  occurring 
i 

D  =  T^of  Different  sequences  in  the  scan 

In  the  case  of  the  instrument  scan.  entropy  has  the  units  of 
bits/sequence  and  provides  a  measure  of  the  randomness  (or  orderliness)  of 
the  scanpath.  The  higher  the  entropy,  the  more  disorder  is  present  in  the 
scan.  The  maximum  possible  entropy  is  constrained  by  the  experimental 
conditions  (see  below).  The  entropy  measure  uses  the  same  probabilities 
which  are  present  in  transition  matrices,  but  it  yields  a  single,  more 
compact  expression  for  the  overall  behavior  of  the  probabilities  rather 
than  presenting  them  each  individually.  This  method  appears  to  afford  some 
generality  and  has  been  the  focus  of  our  recent  efforts. 

To  implement  this  method,  each  of  the  instruments  to  be  examined  was 
given  a  number.  Then  a  sequence  of  these  numbers  was  stored  as  the  pilot 
scanned  the  instrument  panel  together  with  the  dwell  time  for  each 
fixation.  While  sequences  of  up  to  length  4  were  considered  in  preliminary 
analyses,  the  most  detailed  study  was  made  on  sequences  of  length  2.  The 
remainder  of  the  discussion  here  applies  to  the  results  for  length  2 
sequences.  Details  of  themetfiodolgv  are  given  elsewhere  (Stephens.  1981). 

It  can  be  shown  that  the  observed  entroDV  for  the  instrument  scan  is 

4  ,  "■  * 

related  to  the  total  number  of  fixation  sequences  (L.  defined  with  equation 

7  below)  observed  durina  a  run.  In  order  to  comDare  entroDies  from  the 

r  • 

scans  of  different  pilots  for  different  run  lengths,  each  estimate  of 
entropy  had  to  be  corrected  for  L  and  normalized  to  its  maximum  possible 
value.  Hmax.  Hmax  may  be  calculated  as  follows.  In  the  most  general  case. 

M  instruments  may  be  arranged  in  some  arbitrary  fashion  on  the  cockpit 
panel.  For  a  given  number  of  instruments.  M,  and  sequence  length  N,  the 
maximum  number  of  different  fixation  sequences  is  given  by: 


N-  1 

(6)  Q  =  M(M-l)  =  maximum  number  of  sequences  of  length  N 

The  number  of  bits  required  to  uniquely  encode  all  Q  possible  sequences  is 
log2  Q.  The  magnitude  of  this  latter  number  also  represents  Hmax  of  the 
visual  scan  for  the  number  of  instruments  an  sequence  length  being 
considered.  For  example,  with  7  instruments  the  value  of  Q  for  sequences 
of  2  instruments  is  56  which  yields  a  corresponding  Hmax  =  5.8. 

The  normalized  value  of  H  may  then  be  calculated  from: 
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Hmax 

(7)  Hcorr  =  Ho  * - 

Log  L 
2 

where 

L  =  R-N  +  l  =  number  of  sequences  in  a  run 
R  =  number  of  fixations  in  a  run 
N  =  sequence  length  (N  =  1,2.3.  or  4) 


While  entropy  should  help  to  explain  the  orderliness  (or  lack  thereof) 
of  the  scanning  pattern,  the  development  presented  up  to  this  point  does 
not  include  the  fact  that  the  dwell  time  for  each  fixation  is  different. 
From  the  preliminary  results  on  instrument  dwells,  it  appears  rather  clear 
that  dwell  times  can  be  markedly  affected  during  high  mental  loading.  In 
order  to  include  the  effect  of  time  in  our  measure,  a  term  for  entropy  rtc.'+e. 
was  defined  as: 

(8)  Hrate  =  Ho/t 

where  Ho  is  the  entropy  for  the  system  given  by  7  and  t  =  smallest  interval 
in  which  a  transition  may  occur. 

In  practice,  the  calculation  of  Hrate  was  an  average  value  given  by 
the  following: 


0 

(9)  Hrate  Hcorr  /DT 

ava  i  =  l  i  i 

where 

Hcorr  =  Normalized  entropy  for  ith  sequence 
i 

DT  =  Average  Dwell  time  for  ith  sequence 
i 

D  =9^of  different  fixation  sequences 

It  is  helpful  to  estimate  the  maximum  value  which  Hrate  might  assume. 
This  may  be  calculated  using  the  maximum  for  entropy  determined  above 
together  with  dwell  time  statistics  for  the  various  instrument  sequences  in 
the  scan.  While  it  is  possible  for  pilots  to  make  rather  rapid  glances 

(with  dwell  times  of  100  msec  or  less)  at  their  instruments  (Harris  and 

Christhilf,  1980)  a  fixation  rate  this  high  (10  fixations/sec)  rapidly 
leads  to  oculomotor  fatigue.  A  morerealistic  average  value  is  probably 
about  2  fixations/sec  or  less  for  a  long  period  of  instrument  scan  (sav  > 

10  sec). 

Using  0.5  sec/look  (2  fixations/sec)  as  the  average  dwell  interval, 
the  maximum  entropy  rate  for  sequences  of  length  2  is  calculated  to  be 

Hrate  =  5. 8/0. 5  *  2  fixations/seq .  =  6  bits/sec 
max 

This  number  represents  an  upper  bound.  Since  we  suspect  that  the  pilot 

must  have  some  regularity  in  his  or  her  scan,  the  numbers  we  would"  expect 
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to  obtain  under  actual  flight  conditions  will  probably  be  lower.  The 
observed  average  Hrate  for  the  current  experiments  was  on  the  order  of  1 
bit/sec.  A  tendency  to  stare  under  increased  load  should  be  reflected  by 
decreased  entropy  and  increased  fixation  times  making  Hrate  tend  toward 
lower  values  under  such  conditions.  Figure  4  plots  Hrate  vs  number  Task 
Difficulty  for  all  pilots  except  12  and  8. 


Figure  4.  ENTROPY  RATE  ON  LENGTH  2  SEQUENCES  vs. 
IMPOSED  TASK  DIFFICULTY 


A  trend  toward  lower  entropy  rate  with  higher  task  difficulty  may  be  seen. 

A  two-way  analysis  of  variance  was  performed  for  the  entropy  rate  data  from 
nine  pilots  on  levels  of  task  difficulty  and  between  subiects.  F  tests 
allowed  reiection  of  two  null  hypotheses:  equality  of  mean  Hrate  at  all 
loading  levels  (p  <  0.01)  and  equality  of  mean  Hrate  between  subiects  (p  < 
0.01).  All  six  combinations  of  level  differences  in  mean  Hrate  were  found 
to  be  statistically  significant  (T-test  p  <  0.05).  Thus  Hrate  was  chosen 
to  map  from  scanning  behavior  into  task  difficulty  (i.e,  workload). 

The  model  used  expresses  Hrate  as  an  exponential  function  of  TD. 

(10)  Hrate  =  0.9279  EXF(-TD) 

This  equation  was  obtained  via  a  regression  analysis  based  on  the  data  from 
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seven  of  the  pilots  with  a  coefficient  of  determination.  R-souared.  = 
97.3%.  This  equation  may  be  solved  for  task  difficulty  withthe  following 
results: 

(11)  TD  =  -<0.06  +  In  Hrate). 

This  expression  can  then  be  used  to  predict  the  level  of  TD  for  a  new 
subiect  under  the  conditions  of  the  experiment  reported  here. 

Model  Development  and  Verification 

One  of  the  maior  goals  of  this  work  was  the  development  of  a  model 
relating  performance,  skill,  and  mental  workload.  The  ultimate  goal  is  the 
prediction  of  performance  given  estimates  for  skill  and  scanning 
parameters.  A  model  relating  performance,  skill,  and  mental  workload  may 
be  postulated  from  the  empirical  relationship  shown  in  figure  1. 
Construction  of  the  model  should,  in  fact,  aid  in  determino  whether  such 
empirical  expressions  are  valid.  The  model  chosen  was  an  exponential  form: 


2 

(12)  P  =  PCO)  -  EXP((TD/Skill)  ) 

This  equation  may  be  rearranged  as  follows: 

2 

(13)  EXP <<TD /Skill)  )  =  P(0)  -  P 

which  states  that  the  exponential  term  is  equal  to  the  difference  in  te 
performance  at  the  no-loading  level  P<0)  and  the  performance  at  the  present 
level  of  mental  loading  P.  Using  the  values  for  the  level  of  skill  and 
task  difficulty  calculated  in  equations  4  and  11  respectively,  the  left 
hand  side  of  the  equation  may  be  computed.  The  right  hand  side  of  the 
equation  must  be  expressed  in  terms  of  measurable  performance  indicators. 

Expanding  the  right  side  of  (13)  yields 

(14)  P(0)  -  P  =  a0#TP<0>  -  ^tTP)  +  b(RMS/GS<0)  -  RMS/GS) 

+  c(RMS/LOC(0)  -  RMS/LOC)  +  d(%PWR/GS<0)  -  "AFWR/GS) 

+  e(%PWR/LOC<0)  -  %PVR/LOC> 


A  multiple  regression  analysis  was  then  performed  on  equation  13  using 
values  for  each  of  these  measures  recorded  during  the  experiments. 

The  data  from  seven  pilots  was  used  for  model  development,  while  that 
from  three  other  subiects  was  used  for  model  verification.  One  pilot's 
performance  data  was  discarded  due  to  equipment  malfunction. 

The  results  of  the  first  attempt  at  regressin  indicated  that  the 
coefficient  of  the  %PWR/LOC  term  could  not  be  differentiated  from  zero 
based  on  a  Student's  T-test.  This  variable  was  eliminated  from  equation  13 
and  the  analysis  was  repeated.  This  regression  yielded  non-zero  values  for 
the  coefficients  a  through  d.  and  included  a  constant  term.  The  resulting 
equation  was: 
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2 

(15)  EXP((TD/Skill)  )  =  1.4483  +  0.0351^TP(0>  -#TP) 

+  0.1765<fWS/GS<0>  -  RMS/GS) -  0.0366(RMS/LOC(0>  -  RMS/LOC) 
+  0.0377(%FWR/GS (0)  -  %PWR/GS) 

This  analysis  had  an  R  squared  value  of  7S.6  percent  and  an  F-ratio  of 
12.28  (p  <  0.01).  The  coefficients  determined  for  15  may  now  be  used  in 
equation  3  which  becomes 

(16)  P  s  1.4483  +  0.0351^TP)  +  0 . 17G5<RMS/GS) 

-  0.0366(RMS/LOC)  +  0 .0377  (%PWR /GS) . 

These  coefficients  provide  the  relative  weightings  for  each  of  the 
performance  terms  but  they  need  to  be  scaled  in  order  to  provide  the  proper 
characteristics  for  the  equation.  If  each  of  the  terms  were  at  their 
maximum  value,  that  is  100  percent,  then  the  combined  performance  measure 
should  also  equal  100  percent.  However,  using  the  coefficient  this  100 
percent,  each  coefficient  must  be  multiplied  by  100./22.72  =  4.40.  The 
modified  performance  equation  becomes: 

(17)  P  =  6.3750  +  0.1545^TP)  +  0.77G9(RMS/GS)  -  0. 16 1 1  (RMS/LOC) 

+  0.1659(%FWR/GS) 

A  plot  of  this  fuction  versus  the  task  difficulty,  obtained  from  equation 
11,  is  provided  in  Figure  5. 

It  was  hoped  that  these  curves  would  resemble  those  given  in  the 
hypothetical  plot  in  Figure  1  and  for  some  of  the  pilots,  a  general  overall 
downward  trend  is  present.  Even  though  the  curves  do  not  match  the 
hypothetical  ones  exactly,  there  are  some  common  features  between  them. 
First  of  all.  the  curve  for  the  lowest  skilled  pilot  7  is  seen  to  decrease 
much  more  rapidly  than  the  curves  forthe  more  highly  skilled  pilots  (  3, 

11;  the  two  points  for  3  are  for  the  third  and  highest  levels  of  mental 
loading  respectively). 

To  test  this  model's  value  as  a  predictive  tool,  the  data  from  three 
subiects  not  included  in  the  model  determination,  were  substituted  into 
equation  17  and  plotted  versus  perceived  task  difficulty  in  Figure  6. 

Pilots  12,  8.  and  1G  produce  some  interesting,  if  not  consistent 
results.  The  three  points  of  pilot  12.  and  pilot  1G  are  for  the  second, 
third,  and  highest  loading  levels.  All  three  pilots  show  a  net  decrease  in 
performance  between  their  lowest  and  highest  task  difficulties  even  though 
they  accomplished  this  decrease  in  very  different  ways.  Pilot  8  appears 
to  be  the  closest  to  the  theoretical  model  with  his  sharp  decrease  in 
performance  over  a  very  small  task  difficulty  increase.  Pilot  1G,  on  the 
other  hand,  appears  to  be  decreasing  at  an  exponentially  decreasing  rate  as 
opposed  to  the  model  which  predicts  reasing  performance  at  an 
exponentially  increasing  rate.  Pilot  12  increases  performance  sharply 
between  his  second  and  third  runs  and  then  decreases  iust  as  sharply 
between  the  third  and  fourth  runs. 

Since  the  choice  of  the  exponential  model  for 

performance/skill/workload  was  arbitrary,  two  other  forms  for  the  model 
were  also  examined.  These  were  circular  and  linear  models  and  neither  was 
as  good  at  fittino  the  data  as  the  exponential  and  hence  were  abandoned. 


TASK  DIFFICULTY  (FROM  EQ  10) 

Figure  5.  Combined  performance  (from  model)  perceived  task 
difficulty  for  7  pilots  used  in  model  development 
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TASK  DIFFICULTY  (FROM  EQ  10) 

Figure  6.  Combined  performance  vs.  task  difficulty  for  3  test 

248  cases  of  model 


The  models  described  here  are  still  under  development  and  work  is  in 
progress  to  repeat  the  experiments  described  here  and  to  apply  this 
methodology  to  other  instrument  flight  scenarios. 

Summary 

This  paper  presents  some  of  the  findings  frm  a  set  of  experiments 
designed  to  explore  the  relationship  between  performance,  skill,  and  visual 
scanning  behavior  of  aircraft  pilots  under  varying  levels  of  mental 
workload.  Instrument  fixations  were  recorded  as  a  group  of  pilots  with 
widely  varying  levels  of  skill  simultaneously  performed  a  constant 
instrument  flight  task  and  a  verbally  presented  loading  task  with  4 
discrete  levels.  Initial  results  indicate  a  tendency  of  lesser  skilled 
pilots  to  stare  at  the  primary  instrument  as  loading  is  increased  and  to 
alter  the  frequency  of  usage  of  different  scan  paths.  Skilled  pilots 
demonstrated  much  less  change  on  both  of  these  measures. 

A  maior  finding  of  the  research  suggests  that  under  relatively 
constant  instrument  flight  conditions  the  entropy  rate  of  the  visual  scan 
path  may  be  a  useful  measure  of  the  level  of  mental  workload  induced  by  a 
constant  rate  verbal  task.  This  measure  of  workload  was  combined  with 
independent  estimates  of  performance  on  the  piloting  and  verbal  tasks  and 
of  pilot  skill.  An  exponential  model  relating  these  factors  was  developed 
and  has  undergone  preliminary  tests.  The  model  helps  provide  insight  on 
the  intimate  connections  between  a  particular  workload  measure  and  operator 
skill  and  performance  strategy. 
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Question  a. 

Considerable  research  along  similar  lines  was  done  under  ONR/NASA 
sponsorship  several  years  ago  at  STI  (e.g.  see  NASA  CR  1569  and  ONR 
reports  "STI  TR  163-1  and  183-  ) .  Invokes  control  theory  analysis 

to  show  that  scan  patterns  are  not  completely  random,  but  have  pre¬ 
dictable  (explainable)  correlations  with  controlled  element  and  task 
demands.  (Data  show  similar  effects  as  yours). 

Answer  a. 

There  certainly  are  some  interesting  parallels  between  our  work  and 
earlier  studies  at  STI  as  in  NASA  CR  1569.  Both  efforts  reveal  an 
observable  change  in  scanning  behavior  with  varying  task  difficulty. 

The  experimental  conditions  are  somewhat  different,  however,  in  that 
the  STI  work  uses  a  "critical  side  tracking  track"  which  requires  an 
alteration  in  the  scan.  Our  method  (verbal  task  if  varying  difficulty) 
does  not  in  itself  require  an  altered  scan  path  for  its  successful 
performance.  As  the  critical  task  difficulty  increases  the  swell 
times  become  shorter.  For  increased  verbal  loading  in  our  experiments, 
the  dwell  times  become  longer. 

While  these  two  findings  are  not  directly  comparable,  they  do  point 
out  the  potential  utility  of  instrument  scan  in  the  measurement  of 
behavior  of  pilots  and  the  need  for  great  care  in  the  interpretation 
of  scanning  data  within  the  context  of  a  particular  experiment. 


Question  b. 

(a  "nit")  Why  use  the  arcane  term  "entropy"  and  'V2 —  "  rate  when  the 
current  term  (circa  1960*s  and  on)  is  "transinformation  index"  and 
rate  (e.g.  used  by  Ames  references  since  1960's)? 

Answer  b. 

The  use  of  the  word  "information"  would  be  misleading  in  the  context 
of  our  experiments  since  we  do  not  currently  attempt  to  quantify  the 
amount  of  information  the  pilot  is  obtaining  from  his  displays. 
Rather,  we  are  concerned  for  the  moment  only  with  the  orderliness  of 
the  scan  pattern.  The  method  used  to  quantify  the  order  in  the  scan 
was  the  mathematical  form  of  entropy  as  presented  in  the  original 
works  on  information  theory  (Shannon  &  Weaver,  1949).  Entropy  seems 
a  clear  enough  term;  "transinformation"  on  the  other  hand  suggests  a 
broader  meaning  than  we  intend  in  our  work  reported  here. 
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HELICOPTER  PILOT  PERFORMANCE  AND  WORKLOAD  AS  A 
FUNCTION  OF  NIGHT  VISION  SYMBOLOGIES 


E.  James  Hartzell 


U.S.  Army  Aeromechanics  Laboratory 
Moffett  Field ,  CA 


Abstract 


In  the  near  future  military  helicopter  pilots  will  be  required  to 
fly  night  time  nap-of-the-earth  (NOE)  missions  using  helmet- 
mounted  displays  of  their  flight  environment.  These  displays 
will  be  composed  of  an  infrared  video  representation  of  the  out¬ 
side  world  and  flight  control  information,  in  a  symbology  format, 
superimposed  on  the  visual  scene.  The  piloting  task  under  such 
conditions  must  be  classed  as  extremely  demanding  in  terms  of 
performance  requirements,  safety,  and  workload.  A  study  was  un¬ 
dertaken  to  investigate  several  human  factors  questions  of  man- 
machine  integration  mediated  through  such  displays  and  symbolo¬ 
gies.  A  full  six-degree-of-freedom  motion  simulation  of  an  ad¬ 
vanced  U.S.  Army  helicopter  flying  a  night  NOE  scenario  was  con¬ 
ducted.  The  dynamic  visual  scene  was  obtained  from  a  video  pic¬ 
ture  of  a  terrain  board  and  three-candidate  computer  generated 
flight  control  symbologies  were  video-mixed  with  the  scene.  Six 
experienced  helicopter  pilots  were  employed  as  subjects  and 
trained  to  fly  a  scenario  incorporating  multiple  precision  hover 
maneuvers  which  varied  in  difficulty  and  task  loading.  The  ex¬ 
periment  was  designed  to  assess  pilot  performance,  training  re¬ 
quirements,  and  workload  as  a  function  of  the  three  symbologies. 
Workload  is  essentialy  a  cognitive  function  and  is,  therefore, 
difficult  to  measure  objectively  though  Wierwille  and  Williges 
reviewed  some  23  different  techniques  which  have  been  attempted. 
Of  the  techniques  available,  Time  Estimation  was  chosen  as  the 
most  unobtrusive  and  least  problematic  to  implement.  In  a  combat 
environment,  helicopter  pilots  are  required  to  estimate  the 
period  of  time  that  they  can  safely  remain  exposed  or  the  period 
before  exposure  is  executed.  Time  estimation  periods  were, 
therefore  nested  in  the  current  experimental  scenario  which 
agreed  with  and  supported  external  validity.  The  results  of  the 
time  estimation  technique  used  in  conjunction  with  this  experi¬ 
ment  supported  the  predicted  levels  of  difficulty  designed  into 
the  flight  scenario.  Time  estimation  provided  a  relative  scale 
of  workload  between  the  hover  maneuvers  and  suggests  that  there 
is  no  significant  difference  in  workload  between  the  three  sym¬ 
bology  types  studied. 
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An  Investigation  of  the  Effects  of  an  Isometric 
Side-Stick  Controller  on  Pilot  Workload  for  Helicopter 
Terrain  Flight  (Progress  Report) 


John  C.  Hemingway 
NASA-Ames  Research  Center 
Moffett  Field,  California 

Edwin  W.  Aiken  and  Christopher  L.  Blanken 
Aeromechanics  Laboratory 

U.S.  Army  Research  and  Technology  Laboratories  (AVRADCOM) 

Ames  Research  Center 
Moffett  Field,  California 

Abstract 

The  design  and  tentative  results  of  a  moving-base  simulator  experiment 
conducted  to  assess  the  effects  on  pilot  workload  of  replacing  one  or  more  of 
the  standard  helicopter  controllers  with  an  integrated  isometric  side-stick 
controller  are  described.  Using  the  NASA-Ames  Vertical  Motion  Simulator, 
various  low-altitude  helicopter  maneuvers  typical  of  daytime  terrain  flight 
are  simulated  to  investigate  the  effects  of  parameters  such  as  the  number  of 
axes  controlled  through  the  .isometric  device,  the  level  of  stability  and  con- 
tol  augmentation  employed,  and  the  demands  of  the  primary  task  itself.  The 
experimental  data  include  certain  objective  measures  of  pilot  workload: 
power  spectral  density'  analyses  of  control  activity  provide  a  measure  of 
physical  workload;  mental  workload  is  assessed  using  a  secondary  task  based 
upon  the  Sternberg  item  recognition  technique.  This  task,  presented  visually 
through  the  head-up  display,  is  a  memory  scanning  task  requiring  letter 
recognition  and  a  verbal  positive  or  negative  response  to  each  displayed  letter. 
Reaction  time  and  error  data  are  collected  and  analyzed  to  assess  the  effects 
of  the  primary  experimental  variables.  In  addition  to  these  workload  measures, 
substantiating  performance  and  pilot  evaluation  data  are  collected  and  analyzed. 
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DOUGLAS  AI/rC/tAFT  COMPANY 


Ramona  Gulick 


ABSTRACT 

VALIDATION  OF  PILOT  WORKLOAD  ESTIMATES  UTILIZING  IN-FLIGHT  DATA 

The  purpose  of  this  briefing  is  to  .describe  the  current  capabilities  of  the 
Douglas  Aircraft  Company's  workload  evaluation  techniques.  The  objectives  of 
our  workload  evaluation  studies  are  (1)  to  isolate  design  deficiencies  and 
operational  problems  associated  with  specific  hardware  designs  and  functional 
arrangements  of  equipment  during  initial  stages  of  development,  (2)  to  verify 
that  crew  duty  allocations  and  operational  pcorcedures  do  not  exceed  the 
capabilities  of  individual  crewmen,  and  (3)  to  evaluate  crew  composition. 

The  study  approach  includes  the  development  of  design  mission  scenarios  and  an 
in-depth  evaluation  of  selected  mission  segments  using  task/timeline  analytical 
techniques.  Outputs  of  the  crew  workload  evaluation  program  include:  task 
loadings  of  individual  crew  members,  equipment  interface,  and  body  channels. 
Outputs  also  include  external  vision  availability  and  information  processing 
time. 

The  derivation  of  task  performance  time  estimates  are  based  upon  direct  measure¬ 
ments  recorded  during  mockup  trials,  in-flight  analysis  of  video  tapes,  a  reach 
envelope  model,  and  the  studies  of  other  crew  workload  studies  (Air  Force, 
industry).  Information  processing  time  estimates  were  derived  from  studies/ 
experiments  cited  in  the  literature  and  DAC  laboratory  evaluations  of  electronic 
flight  display  formats. 

The  accuracy  of  task  time  estimates  has  been  validated  through  direct  in-flight 
observations,  including  the  FAA  certification  flight  tests  of  commercial • trans¬ 
port  aircraft  and  micro-motion  analysis  of  time-referehced  video  and  audio 
recordings.  Results  of  these  studies  showed  a  high  correlation  between  estimated 
task  time  in  actual  in-flight  observations. 
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THE  FOCUS  OF  THIS  BRIEFING  IS:  1)  TO  DESCRIBE  DOUGLAS  AIRCRAFT'S 
ANALYTICAL  TECHNIQUES  FOR  FLIGHT  CREW  WORKLOAD  EVALUATION,  2)  TO  EXPALIN 
THE  METHODOLOGY  EMPLOYED  IN  VALIDATING  THE  ANALYSIS,  AND  3)  TO  SHOW  THE 
APPLICATION  OF  THIS  MODEL  TO  DETERMINE  CREW  COMPLEMENT  AND  DESIGN 
VERIFICATION. 

THE  VALIDATION  OF  THE  ANALYTICAL  TASK  TIMES  WAS  DONE  THROUGH  IN-FLIGHT 
TESTING  DURING  THE  DC-9  SUPER  80  CERTIFICATION  PROCESS. 

THIS  ANALYSIS  WAS  APPLIED  ON  OUR  RECENT  C-X  PROPOSAL  TO  THE  AIR  FORCE. 


THE  FOUR  OBJECTIVES  OF  THE  ANALYSIS  WERE: 

•  VALIDATE  THE  ANALYTICAL  METHODOLOGY  THROUGH  IN-FLIGHT  DATA 

»  TO  DETERMINE  THE  WORKLOAD  IMPACT  'OF  VARIOUS  DESIGNS  DURING  THE 
DEVELOPMENT  PROCESS 

•  VERIFY  THAT  OPERATOR  TASK  LOADINGS  IMPOSED  ON  THE  CREW  BY  THE  STATION 
CONFIGURATIONS  ARE  NOT  EXCESSIVE 

•  DETERMINE  THE  FEASIBILITY  OF  THE  TWO  FLIGHT  CREW,  ONE  LOADMASTER 
CONCEPT. 


IN  THE  STUDY  APPROACH,  IT  WAS  NECESSARY  TO  DEVELOP  A  REPRESENTATIVE 
FLIGHT  SCENARIO  TO  EXERCISE  THOSE  FUNCTIONS  CONSIDERED  ESSENTIAL  FOR 
DEMONSTRATING  OPERABILITY  FOR  THE  MINIMUM  CREW. 
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AS  A  BASELINE  FOR  OUR  ANALYTICAL  TECHNIQUES,  A  MISSION  WAS  “FLOWN”  WITH 
ALL  SYSTEMS  OPERATING  NORMALLY  AND  THEN  INTRODUCE  INCREASINGLY  DEGRADED 
MODES  WERE  INTRODUCED  IN  SUBSEQUENT  SCENARIOS  TO  DETERMINE  THEIR  IMPACT 
ON  THE  WORKLOAD--AND  LASTLY— TO  DETERMINE  THE  ACCURACY  OF  THE  TASK  TIME 
ESTIMATES  USED  IN  THE  ANALYTICAL  METHOD,  ACTUAL  IN-FLIGHT  DATA  WOULD  BE 
ANALYZED  FOR  COMPARISON. 


WORKLOAD  IS  DEFINED  AS  THE  RATIO  OF  THE  TOTAL  TIME  REQUIRED  TO  PERFORM 
A  SERIES  OF  TASKS  TO  THE  TIME  AVAILABLE  WITHIN  THE  CONSTRAINTS  IMPOSED 
BY:  THE  MISSION,  AIRCRAFT  PERFORMANCE  CHARACTERISTICS,  AND  CERTAIN 
EXTERNAL  FACTORS  SUCH  AS  ATC  AND  METEROLOGI CAL  CONDITIONS.  THIS  WORKLOAD 
IS  EXPRESSED  AS  A  NUMERICAL  INDEX  WHOSE  MAXIMUM  VALUE  IS  TOO  (ALTHOUGH 
UNDER  SOME  SITUATIONS  THE  INDEX  MAY  EXCEED  100). 

NUMBER  5 


THIS  FLOW  DIAGRAM  ILLUSTRATES  THE  PROCESS  USED  IN  OBTAINING  CREW  WORKLOAD 
INDEX  VALUES.  THE  MISSION  SCENARIO  AND  TASK  ANALYSIS  WITH  THE  ASSOCIATED 
TIMES  ARE  CONSTRUCTED  FROM  THE  PREI  IMINARY  AIR  VEHICLE  DESIGN.  THE  TIME 
AVAILABLE  IS  DETERMINED  BY  THE  MISSION  PROFILE,  AIRCRAFT  PERFORMANCE, 

AND  .ENVIRONMETAL  CONDITIONS.  THE  TIME  REQUIRED  FOR  EACH  TASK  IS 
DETERMINED  BY  USING  A  TASK/TIME  DATA  BANK  THAT  PROVIDES  TIMES  FOR  REACHING 
AND  ACTUATING  VARIOUS  TYPES  OF  CONTROLS  AND  FOR  READING  VARIOUS  TYPES  OF 
INSTRUMENTS  IN  RELATION  TO  A  SPECIFIC  TASK  SEQUENCE.  THIS  METHODOLOGY 
WILL  BE  EXPLAINED  LATER.  THESE  TIMES  ARE  BASED  ON  THE  CREW  STATION 
CONFIGURATION  AND  OPERATING  PROCEDURES. 
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THE  WORKLOAD  COMPUTER  PROGRAM  CATEGORIZES  AND  ANALYZES  THE  DATA  INTO 
VARIOUS  OUTPUTS- -WORKLOAD  FOR  EACH  INDIVIDUAL  CREW  MEMBER,  EXTERNAL 
VISION  AVAILABILITY,  TASK  LOADINGS  FOR  EQUIPMENT  INTERFACE,  THE  VARIOUS 
BODY  CHANNELS,  AND  INFORMATION  PROCESSING. 


SEVERAL  SOURCES  WERE  USED  IN  THE  TASK  TIME  ESTIMATES. 

•  THE  BASIC  SUBTASK  PERFORMANCE  TIMES  CAME  FROM  THE  INDEX  OF  ELECTRONIC 
EQUIPMENT  OPERABILITY  DEVELOPED  BY  THE  AMERICAN  INSTITUTE  FOR  RESEARCH 
OR  AIR  DATA 

•  A  DOUGLAS  DEVELOPED  EMPIRICAL  FORMULA  FOR  ESTIMATING  REACH  TIMES 
AS  A  FUNCTION  OF  DISTANCE 

•  VERBAL  COMMUNICATIONS  WERE  REPEATEDLY  TIMED  FOR  A  REALISTIC  AVERAGE 

6  DIRECT  ACTION  TIME  MEASUREMENTS  WERE  RECORDED  DURING  PROCEDURAL 
TRIALS  IN  THE  CREW  STATION  DEVELOPMENT  MOCKUP 

f  TIME  REFERENCED  VIDEO  RECORDINGS  ACQUIRED  DURING  PREVIOUS  IN-FLIGHT 
MICRO-MOTION  STUDIES  WERE  USED  TO  FURTHER  ADD  REALISTIC  TIME  ESTIMATES 


TIMES  FROM  PREVIOUS  LABORATORY  INVESTIGATIONS  RELATED  TO  VISUAL  PERCEPTION 
DECISION  MAKING  ACTIVITIES  WERE  USED.  AIR  DATA  PROVIDED  INFORMATION 
PROCESSING  TIMES  IN  MANY  AREAS.  TIMES  DEVELOPED  DURING  EXPERIMENTAL 
TESTS  CONDUCTED  AT  DOUGLAS  IN  SUPPORT  OF  THE  ADVANCED  ELECTRONIC  DISPLAY 
FORMAT  DEVELOPMENT  WERE  ALSO  USED. 
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•  THE  PROGRAM  COMPUTES  A  WORKLOAD  INDEX  VALUE  FOR  EACH  INDIVIDUAL 
CREWMEMBER  AND  REPRESENTS  A  COMPOSITE  OF  THE  PHYSICAL  AND  COGNITIVE 
ACTIVITIES  REQUIRED  TO  COMPLETE  THE  TASK  SEQUENCE. 

•  WORKLOAD  VALUES  ASSOCIATED  WITH  SPECIFIC  EQUIPMENT  MAY  BE  TABULATED 
USING  CATEGORY  CODES  SUCH  AS  ATA  CHAPTER  OR  MILITARY  SYSTEMS/SUBSYSTEM/ 
SUBJECT  NUMBERS  TO  ISOLATE  OPERATIONAL  EFFICIENTY  OR  INEFFICIENCY  OF 
THAT  PARTICULAR  TYPE  OF  EQUIPMENT. 

t  THE  TASK  LOADING  IMPOSED  ON  ONE  INDIVIDUAL  BODY  CHANNEL  MAY  BE 
DETERMINED-RIGHT  HAND,  LEFT  HAND,  INTERNAL  VISION,  VERBAL/AURAL 
AND  FEET. 

•  THE  TIME  REMAINING  WHEN  THE  VISUAL  MONITORING  OF  INTERNAL  DISPLAYS 

AND  CONTROLS  IS  NOT  UTILIZED  IS  CONSIDERED  EXTERNAL  VISION  AVAILABILITY 
OR  TIME  AVAILABLE  FOR  EXTERNAL  ENVIRONMENTAL  SCANS. 

•  ESTIMATED  TIME  VALUES  WERE  ALSO  ASSIGNED  FOR  THOSE  TASKS  INVOLVING 
AN  INFORMATION  PROCESSING  COMPONENT. 


A  DEDICATED  FLIGHT  TEST  WAS  MADE  DUPLICATING  AS  CLOSE  AS  POSSIBLE  THE 
SCENARIO  DEVELOPED  FOR  THE  ANALYTICAL  STUDY.  VIDEO  AND  AUDIO  RECORDINGS 
WERE  MADE  AND  ANALYZED  THEN  CORRELATED  WITH  THE  ESTIMATED  TIME  VALUES. 

IN  ADDITION,  THE  FAA  CERTIFICATION  FLIGHTS  MADE  ON  THE  EAST  COAST  TO 
DETERMINE  CREW  COMPLEMENT  WERE  RECORDED  AND  ANALYZED  FOR  FURTHER  VALIDATION. 
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IN  ORDER  TO  OBTAIN  IN-FLIGHT  VIDEO  TAPE  FOR  THE  DATA  BASE  VALIDATION, 

A  DEDICATED  TEST  FLIGHT  WAS  ARRANGED  DURING  THE  DC-9  SUPER  80 
CERTIFICATION  PROCESS.  TWO  CAMERAS--ONE  FOR  DAYTIME  AND  AM  INFRARED 
FOR  NIGHT--WERE  MOUNTED  ON  THE  COCKPIT  BULKHEAD  TO  RECORD  FLIGHT  CREW 
ACTIVITY.  THE  FLIGHT  PROFILE  DUPLICATED  AS  CLOSE  AS  POSSIBLE  THE  SCENARIO 
USED  IN  THE  TASK  TIME  ANALYSIS  STUDY— A  ROUND  ROBIN  LAX— LAX  FLIGHT. 


A  TELEVISION  RECEIVER  WAS  MOUNTED  MIDWAY  IN  THE  CABIN  AREA.  AIRSPEED, 
ALTITUDE,  AND  HEADING  INDICATORS  WERE  INSTALLED  BESIDE  THE  TV  MONITOR 
WITH  A  BOX  DISPLAYING  DIGITAL  READOUTS  OF  THE  TWO  NAVIGATION  AND  TWO 
VHF  COMM  FREQUENCIES  SELECTED  BY  THE  PILOTS.  THE  VHF  RADIOS  WERE  TAPPED 
AND  THE  PILOTS  EACH  WORE  A  CHEST  MICROPHONE  FOR  DISCRETE  MONITORING  OF 
AIR  TO  GROUND  AND  COCKPIT  CONVERSATION.  A  TIME  CODED  VIDEO  TAPE  RECORDER 
SIMULTANEOUSLY  TAPED  BOTH  THE  VIDEO' AND  AUDIO  PORTIONS  OF  THE  FLIGHT 
FOR  FUTURE  ANALYSIS. 


THE  TECHNIQUE  FOR  ANALYZING  THE  VIDEO  TAPE  FOLLOWED  TWO  STEPS: 

•  FIRST  THE  TAPE  WAS  VIEWED  AT  NORMAL  SPEED  TO  DETERMINE  IF  THE  PROCEDURES 
AND  DUTY  ALLOCATIONS  CORRESPONDED  TO  DATA  BASE  ESTIMATES.  ALSO,  IT  WAS 
DURING  THIS  REVIEW  THAT  THE  TASKS  AND  SUBTASKS  COULD  BE  IDENTIFIED  AS 
ONES  WHICH  COULD  BE  ACCURATELY  TIMED. 

•  SECONDLY,  THE  TAPES  WERE  REVIEWED  IN  SLOW  MOTION  AND  STOP/REVERSE/ 

START  MODES  FOR  THE  MICRO-MOTION  ANALYSIS. 
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NUMBER  13 


AN  EXAMPLE  OF  TWO  TASKS  COMPARISON  IS  SHOWN  HERE.  THE  VIDEO  TAPE 
"RETRACT  GEAR"  TASK  IS  TIMED  FROM  THE  START  OF  THE  PILOT'S  CALL  FOR 
"GEAR  UP"  THROUGH  THE  COPILOT'S  MOTIONS  AND  THE  RETURN  OF  HIS  HAND  TO 
REST.  THE  TASK  FROM  THE  DATA  BASE  IS  SUBDIVIDED  INTO  SUBTASKS. 

NUMBER  14 

IN  FACT,  A  COMPARISON  OF  SIXTY-EIGHT  DATA  BASE  AND  THE  CORRESPONDING 
VIDEO  TAPE  TASK  TIMES  INDICATE  THAT  THERE  IS  NO  SYSTEMATIC  TENDENCY 
TO  OVERESTIMATE  OR  UNDERESTIMATE  THE  ACTUAL  TIMES.  IN  FACT,  THERE  WAS 
A  SLIGHT  TENDENCY  TO  OVERESTIMATE  THE  ACTUAL  TIMES  RESULTING  IN  A 
PREDICTED  WORKLOAD  ON  THE  CONSERVATIVE  SIDE. 

NUMBER  15 

STATISTICAL  RESULTS  PLOTTING  THE  ESTIMATED  TASK  TIMES  VERSUS  THE  ACTUAL 
TASKS  TIMES  SHOW  THE  CORRELATION  COEFFICIENT  TO  BE  .848— VERY  HIGH  FOR 
A  VALIDATION  COEFFICIENT. 

«  OF  THE  SIXTY-EIGHT  TASKS,  THE  AVERAGE  TIMES  FOR  THE  ACTUAL  VALUES 
WAS  4.04  SECONDS  ...  FOR  THE  ESTIMATED  4.05  SECONDS 

•  NONE  WERE  OVER  13  SECONDS  DURATION 

•  STANDARD  DEVIATION  OF  ESTIMATED  2.73. . .ACTUAL  2.36 
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TASK  TIME  COMPARISON 
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DATA  BASE  VALIDATION 
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IN  SUMMARY: 


•  THE  MICRO-MOTION  TECHNIQUE  FOR  VIDEO  TAPE  ANALYSIS  PROVIDES  FOR  AN 
ACCURATE  OBJECTIVE  MEASUREMENT  OF  IN-FLIGHT  TASK  TIMES 

•  THE  ESTIMATED  DATA  BASE  TIMES  CORRESPOND  CLOSELY  WITH  THE  MEASURED 
IN-FLIGHT  VIDEO  TAPE  TASK  TIMES 

I  ANALYTICAL  MODEL  PROVIDES  AN  ACCURATE  INDEX  TO  PREDICT  IN-FLIGHT 
WORKLOAD  LEVELS 


THE  PREVIOUS  SLIDES  SHOWED  THE  METHODOLOGY  AND  VALIDATION  RESULTS  FROM 
THE  DC-9  SUPER  80  CERTIFICATION  PROGRAM. 

THIS  SAME  ANALYTICAL  MODEL  WAS  APPLIED  IN  DOUGLAS'  RESPONSE  TO  THE 
AIR  FORCES'  C-X  RFP  IN  WHICH  THE  FOLLOWING  REQUIREMENTS  WERE  ISSUED. 

9  WORKLOAD  REDUCTIONS  SHOULD  BE  ACCOMPLISHED  BY  TASK  ANALYSIS 

ft  VERIFY  THE  INTEGRATION  WITHIN  AND  BETWEEN  FLIGHT  DECK  AND  CARGO 


COMPARTMENT 


THE  AIRDROP  MISSION  SEGMENT  WAS  SELECTED  FOR  THE  INITIAL  CREW  WORKLOAD 
EVALUATION  FOR  THREE  BASIC  REASONS: 

9  IT  PLACES  SUBSTANTIAL  DEMANDS  ON  THREE  FUNCTIONS  IDENTIFIED  PREVIOUSLY 
AS  "MISSION  CRITICAL" 

NAVIGATION 
STATION  KEEPING 
AERIAL  DELIVERY 

•  IT  PROVIDES  AN  OPPORTUNITY  TO  EVALUATE  WORKLOAD  IMPOSED  ON  ALL  THREE 
CREW  MEMBERS  — PILOT,  COPILOT,  AND  LOADMASTER. 

9  THE  AIRDROP  MISSION  IS  ALSO  CONSIDERED  BY  CURRENT  MAC  CREWS  TO  BE  A 
HIGH  WORKLOAD  MISSION. 

NUMBER  19 

IN  ORDER  TO  INSURE  THAT  THE  TASK/TIMELINE  ACCURATELY  REFLECTS  THE  OPERATING 
ENVELOPE  OF  THE  AIRCRAFT,  A  DETAILED  ALTITUDE/TIME  PROFILE  WAS  DEVELOPED 
BASED  ON  THE  ANTICIPATED  PERFORMANCE  CHARACTERISTICS  OF  THE  AIRCRAFT  AND 
MILITARY  PROCEDURES  FOR  A  REPRESENTATIVE  AIRDROP. 

NUMBER  20 

CREW  DUTY  ALLOCATIONS  WERE  CALCULATED  TO  DISTRIBUTE  TASKS  EQUALLY  WHILE 
RETAINING  COMMAND/DECISION  RESPONSIBILITIES: 

•  PRIMARY  PILOT  DUTIES  (SECONDARY  COPILOT  DUTIES) 

•  PRIMARY  COPILOT  DUTIES  (SECONDARY  PILOT  DUTIES) 


9  PRIMARY  LOADMASTER  DUTIES 
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C-X  AIRDROP  MISSION  SEGMENT 
ALTITUDE  -  TIME  PROFILE 
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TIME  (MINUTES) 
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NUMBER  21 


THIS  TABLE  PROVIDES  A  LISTING  OF  THE  INPUT  PARAMETERS  NECESSARY  TO 
CONSTRUCT  A  SAMPLE  TASK— OPEN  CARGO  DOOR  AND  RAMP.  THIS  IS  ONLY  ONE 
OF  THE  TASKS  WHICH  OCCURS  DURING  THE  MILESTONE  BOUNDED  BY  THE  PIP  (OR 
PRE-INITIAL  POINT)  AND  THE  IP  (INITIAL  POINT).  THE  TASK  IS  FURTHER 
SUBDIVIDED  INTO  SUBTASKS  WHICH  ARE  INDIVIDUAL  AIRCREW  ACTIONS  REQUIRED 
TO  PERFORM  THE  TASK. 


NUMBER  22 

INSPECTION  OF  THE  GRAPHIC  OUTPUT  OF  TASK  LOADING  BY  MILESTONE  REVEALS 
A  GENERALLY  LOW  WORKLOAD  FOR  BOTH  THE  PILOT  AND  COPILOT.  THE  MAXIMUM 
WORKLOAD  INDEX  FOR  ANY  MILESTONE  DOES  NOT  EXCEED  SIXTY. 


NUMBER  23 


THE  WORKLOAD  PROFILES  PRESENTED  PREVIOUSLY  ASSUME  THAT  ONLY  THE  CONTROL 
STICK  STEERING  MODE  OF  THE  AUTOFLIGHT  CONTROL  SYSTEM  (AFCS)  IS  UTILIZED. 
A  SECOND  ANALYSIS  WAS  CONDUCTED  TO  ASSES  THE  IMPACT  OF  USING  THE  FULL 
AFCS  ON  THE  WORKLOAD  LEVELS  OF  THE  PILOT. 

RESULTS  SHOW  A  SIGNIFICANT  DECREASE  IN  WORKLOAD  LEVELS  RESULTING  FROM 
FULL  USE  OF  THE  AUTOFLIGHT  CONTROL  SYSTEM  AND  DEMONSTRATE  THE  COMPARISON 
CAPABILITIES  OF  THE  PROGRAM. 
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INPUT  PARAMETERS  FOR  A  REPRESENTATIVE  TASK 
AIRDROP  MISSION  SEGMENT 
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FLIGHT  CREU  MEMBER  TASK  LOADING 
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28ain  865  tod  3k  825  lOain  pip  ip  1 
MILESTONES  -  28  MIN  WARNING  TO  PAU 


CSS  UERSUS  A P  NODE  COMPARISON 
Pilot 
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20r1r  065  tod  3k  025  10*in  pip  ip  1*4 
MILESTONES  -  20  MINUTE  WARNING  TO  PAU 


NUMBER  24 


THIS  TABLE  SUMMERIZES  THE  OVERALL  TASK  LOADINGS  ASSOCIATED  WITH  EACH 
FUNCTIONAL  EQUIPMENT  CATEGORY  AND  HIGHLIGHTS  POSSIBLE  AREAS  FOR  AUTOMATION 
OF  CERTAIN  SYSTEMS  IF  AN  EXCESSIVE  WORKLOAD  EXISTS. 


THE  RESULTS  OF  THIS  ANALYSIS  REVEAL  AN  EVEN  DISTRIBUTION  OF  TASKS  WITH 
THE  WORKLOAD  INDEX  FOR  THE  VARIOUS  CREWMEMBERS  WELL  WITHIN  AN  ACCEPTABLE 
RANGE. 

THE  CREW  STATION  CONFIGURATION  ALLOWED  FOR  FLEXIBILITY  IN  DUTY  ALLOCATIONS 
SO  AS  TO  INSURE  MISSION  ACCOMPLISHMENT  WITH  THE  PROPOSED  MINIMUM  CREW 
COMPLEMENT. 

THEREFORE,  WE  FEEL  THROUGH  THE  VALIDATION  OF  DOUGLAS'  WORKLOAD  MODEL 
WE  CAN,  WITH  SOME  CERTAINTY,  PREDICT  IN-FLIGHT  WORKLOAD  LEVELS. 
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DOCUMENTATION  0  00 

CONTROLS  A  DISPLAYS  1*58 


THE  ELUSIVE  GOAL  OP  MEASURING 
PILOT  WORKLOAD  IN  GENERAL  AVIATION 

Earl  S.  Stein  John  Fabry  Bruce  Rosenberg 
Federal  Aviation  Administration  Technical  Center 
The  magnetic  attraction  of  people  to  aviation  has  existed 
since  the  beginning  of  manned  flight.  This  is  particularly  true 
in  general  aviation  where  the  majority  of  pilots  are  not  only 
unpaid  but,  in  fact,  pay  their  own  expenses.  For  these 
individuals  and  others  in  aviation,  potential  changes  in  aircraft 
cockpits  must  clearly  demonstrate  advantages  in  excess  of  costs. 

Given  a  highly  complex,  multidimensional  stimulus/task 
environment,  how  does  one  determine  pilot  workload  and  perform¬ 
ance?  These  two  global  labels  are  the  basic  elements  of  what 
could  be  called  overall  "pilot  effectiveness."  It  has  been  very 
apparent  from  the  literature  in  aviation  and  human  factors 
psychology  that  usable  systematic  measures  are  not  currently 
available  for  general  aviation.  The  Federal  Aviation  Adminis¬ 
tration  (FAA)  Technical  Center  in  Atlantic  City,  New  Jersey,  has  a 
requirement  to  evaluate  the  impact  of  both  new  and  altered  cockpit 
systems.  The  human  impact  of  these  changes  must  be  quantified  in 
terms  of  "pilot  effectiveness." 

A  detailed  examination  of  the  literature  in  human  workload 
has  led  to  certain  conclusions.  No  one  questions  the  complexity 
or  multidimensional  nature  of  workload.  However,  models  have  been 
largely  design-specific  with  each  experimenter  tailoring  his/her 
theoretical  model  to  meet  his/her ' immediate  needs  and  many  models 
stress  the  atomization  of  workload  into  more  "basic  elements," 
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such  as  mental  and  physical  components.  Measurement  methods  have 

t 

included  physiological  techniques  through  performance  indices  and 
psychological  approaches.  The  most  popular  method  by  far  has  been 
the  postflight  questionnaire.  This  technique  is  easy  to  administer 
but  relies  heavily  on  pilot  memory  for  accuracy.  Given  all  the 
effort  expended,  there  has  been  a  surprising  lack  of  success  in 
aviation  workload  measurement,  and  it  has  not  been  well  linked  to 
any  clear  definition  of  task  difficulty. 

The  literature  in  the  area  of  pilot  performance  dates  well 
back  into  World  War  II  and  beyond.  It  has  largely  been  focused  on 
the  graining  environment  where  cost-benefit  assessment  is 
constantly  emphasized.  The  most  frequently  used  technique  has 
been  instructor/pilot  ratings.  While  these  can  be  made  reasonably 
objective  and  valid,  often  they  are  not,  and  depending  heavily  on 
face  validity,  which  means  essentially  how  good  they  look  to  the 
people  who  must  use  them.  Individual  instructor/pilot  interests 
and  biases  can  reduce  measurement  reliability  considerably.  Such 
techniques  may  serve  a  valuable  need  as  training  diagnostic 
indicators,  but  they  lack  precision  to  identify  what  may  be  subtle 
changes  in  pilot  behavior  induced  by  cockpit  systems  alterations. 

In  the  past  few  years,  there  has  been  a  new  trend,  primarily  in 
military  aviation,  to  employ  a  computerized  data  collection  system 
to  evaluate  performance.  Automated  performance  measurement  (APM) 
allows  for  the  collection  of  aircraft  state  and  control  input 
variables  at  a  very  high  sampling  rate.  Data  can  be  accurately  and 
objectively  recorded.  The  method  assumes  that  inferences  can  be 
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be  made  about  pilot  performance  based  on  how  the  aircraft  is 
flying  and  on  the  nature  and  intensity  of  control  inputs. 

The  FAA  Technical  Center  has  developed  its  own  workload  and 
performance  models.  While  these  models  are  not  completely  atheo- 
retical,  pragmatism  rather  than  theory  has  been  emphasized.  The 
workload  model  assumes  that  if  you  ask  a  pilot  how  hard  he/she  is 
working,  he/she  will  be  able  to  tell  you.  This  is  provided  that 
the  question  is  properly  timed  and  formatted.  The  model  further 
assumes  that  pilots'  responses  will  weigh  the  input  of  all 
stressors  which  currently  effect  his/her  behavior.  No  attempt 
will  be  made  to  force  the  pilot  to  atomize  workload  into  multiple 
catagories  since  this  would  make  the  workload  response  requirement 
more  intrusive.  The  final  assumption  made  by  the  model  is  that 
the  best  time  to  measure  workload  is  during  the  flight  itself. 

The  pilot  performance  model  stresses  automated  performance 
measurement.  Performance,  unlike  workload,  can  be  observed 
directly  and  can  be  measured  unobtrusively.  Here  an  atomistic 
approach  will  be  taken.  Subject  matter  experts  have  identified 
eight  segments  of  a  normal  general  aviation  flight.  These 
include:  1.  takeoff,  2.  climb,  3*  en  route,  4.  holding, 

5.  descent,  6.  initial  approach,  7»  final  approach,  and  8.  missed 
approach.  Each  segment  contains  specific  variables  which  the 
subject  matter  experts  felt  were  critical  for  evaluating 
performance.  For  example,  the  takeoff  segment  contains  heading, 
airspeed,  manifold  pressure,  rpm,  pitch,  and  bank  angles.  Each 
variable  has  an  assigned  ideal  value  and  acceptable  limits  around 
that  value.  The  model  assumes  that  the  results  for  all  variables 
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can  be  summed  to  provide  a  segment  total  score  and  that  segment 
scores  can  be  pooled  for  a  total  flight  performance  score,  which 
will  be  called  the  pilot  proficiency  index  or  PPI.  Since  the 
construction  of  the  PPI  has  been  analytical,  empirical 
verification  will  be  essential  before  it  can  be  put  to  any 
practical  use. 

Two  workload  studies  have  been  completed  already.  The  first 
study  employed  a  nonflying,  two-axis,  critical  tracking  task. 
Twenty-four  people,  of  whom  12  were  pilots,  were  asked  to  maintain 
a  point  of  light  in  the  center  of  a  cathode-ray  tube  (CRT)  display 
using  a  joystick  control.  This  task  has  been  compared  to 
attempting  to  balance  a  ball  on  the  end  of  a  broomstick.  The 
further  one  strays  from  the  center,  the  more  difficult  it  becomes. 
The  driving  force  behind  the  display  was  an  analog  computer  which 
could  alter  the  difficulty  level  along  a  continuum.  Each 
participant's  maximum  ability  was  measured  and  then  difficulty 
levels  were  tailored  as  proportions  of  their  own  maximums.  These 
proportions  included  four  levels:  .25,  .50,  .75,  and  1.00. 

Trials  were  run  in  a  counterbalanced  order.  During  each  4  1/2 
minute  trial,  a  query  tone  was  sounded  every  minute.  The  partic¬ 
ipant  was  asked  to  press  a  button  from  1  (very  easy)  to  10  (very 
hard)  which  best  described  how  hard  he/she  was  working.  Results 
indicated  that  participants  were  willing  and  able  to  make  the 
workload  evaluations  during  primary  task  performance.  Their 
workload  values  were  directly  related  to  the  difficulty  level 
assigned  for  any  given  trial.  Response  latency,  however,  was  not 
related  to  difficulty. 
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A  second  workload  study  employed  a  Singer-Link  General 
Aviation  Trainer  (GAT-2)  which  simulated  a  light  twin  cabin  class 
aircraft  (Cessna  421).  Three  qualitatively  different  levels  of 
flight  difficulty  were  assigned.  Plight  difficulty  was  controlled 
by  manipulating  clearance  complexity,  air  traffic  control 
complexity,  air  turbulence,  and  an  inflight  emergency.  Twelve 
pilots  each  flew  three  flights  in  a  counterbalanced  order.  They 
were  instructed  to  evaluate  their  inflight  workload  using  a  10 
pushbutton  scale  mounted  in  a  box  below  the  throttles.  During 
flight,  their  responses  were  requested  every  minute  using  both  a 
tone  and  a  light.  The  tone  lasted  only  one-fifth  of  a  second 
while  the  light  (a  small  LED  on  the  switchbox)  stayed  on  until  a 
button  was  pressed.  Results  indicated  that  workload  responses 
increased  significantly  in  direct  relationship  to  the  difficulty 
level.  Response  delays  also  increased  with  difficulty  but  only 
between  the  easiest  and  the  moderately  difficult  flights.  There 
was  no  difference  between  the  moderate  and  most  difficult  flights 
in  terms  of  response  delay.  One  explanation  for  this  is  the 
possibility  that  as  difficulty  increased,  the  participant's 
attention  became  more  focused  on  workload  and  on  their  evaluations 
of  it.  The  second  study  clearly  demonstrated  that  pilots  were 
able  to  make  workload  discriminations  during  simulated  flight. 

The  results  of  postflight  questionnaire  indicated  that  pilots  did 
not  find  using  the  pushbuttons  to  be  overly  obtrusive. 

While  workload  work  will  continue,  performance  research  is 

just  beginning  at  the  PAA  Technical  Center.  An  initial  test  of 

the  PPI  concept  will  involve  flying  two  diverse  groups  in  the 
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Center's  GAT-2.  These  groups  sill  include  high-time,  professional 
pilots  and  low-time,  just  qualified  instrument/multiengine  pilots. 
Assuming  that  the  PPI  can  discriminate  between  these  two  groups, 
additional  tests  will  be  accomplished  to  validate  the  method. 

While  in  the  short  run,  APM  will  be  confined  to  simulation,  long¬ 
term  possibilities  could  include  airborne  flight  tests  using 
state-of-the-art  microcomputer  techniques  for  data  collection. 

An  airborne  workload  data  collection  system  has  already  been 
constructed  in  prototype.  This  will  be  evaluated  based  on  the 
availability  of  flight  time.  It  is  likely  that  it  will  be  used  in 
conjunction  with  other  projects. 

Techniques  for  measuring  workload  and  performance  in  general 
aviation  are  under  active  development.  The  goal  is  to  establish 
the  tools  necessary  in  order  to  provide  timely  and  accurate 
information  concerning  the  effects  of  systems  changes  on  pilot 
behavior.  Only  through  active,  empirical  research  can  such  tools 
be  developed. 
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SUBJECTIVE  WORKLOAD  ASSESSMENT  TECHNIQUE 
Gary  B.  Reid,  F.  Thomas  Eggemeier 
and  Clark  A.  Shingledecker 


Subjective  assessment  techniques  are  often  employed  in  flight  testing 
and  operational  test  and  evaluation.  The  acceptability  of  these  measures  is 
best  demonstrated  by  the  widespread  use  and  utility  of  the  Cooper-Harper 
scale  for  assessing  aircraft  handling  qualities. 

We  in  the  Workload  and  Ergonomics  Branch  of  the  Air  Force  Aerospace 
Medical  Research  Laboratory  are  engaged  in  a  program  of  research  to  develop 
a  comprehensive  battery  of  tests  for  assessing  pilot  workload.  A  consider¬ 
able  part  of  this  program  is  devoted  to  the  development  and  validation  of 
physiological  measures,  behavioral  measures,  and  systems  performance 

4 

measures.  Even  though  the  primary  emphasis  of  our  program  is  not  the 
development  of  subject  measures  we  felt  that,  in  order  for  our  test 
battery  to  be  complete,  we  needed  a  systematically  developed  and  validated 
subjective  measure  that  is  generally  applicable  and  widely  accepted. 

A  thorough  literature  review  of  subjective  measures  revealed  that 
precisely  what  we  wanted  does  not  currently  exist.  While  subjective 
measures  are  frequently  used  in  workload  assessment,  they  usually  are 
designed  for  a  specific  application.  The  other  category  of  readily  avail¬ 
able  measures  includes  measures  like  the  Cooper-Harper  and  SOMA  scales. 

These  scales  assess  handling  qualities  and  system's  operability  respectively 
and,  as  such,  are  focused  on  systems  evaluation  with  workload  as  a  component. 
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The  measure  we  desire  should  be  designed  specifically  to  assess  workload. 
Within  the  context  of  our  battery  this  measure  is  conceived  as  being  less 
precise  than  some  of  the  other  proposed  measures  (e.g.  cortical  evoked 
response)  but  should  be  precise  and  sensitive  enough  to  quantify  the 
existence  of  high  workload.  Also,  this  scale  should  be  simple  to 
administer  so  that  applications  such  as  operational  test  and  flight  test 
are  possible.  Since  none  of  the  currently  existing  scales  that  we 
reviewed  possessed  these  attributes,  we  initiated  an  effort  to  develop 
our  own  subjective  measure  which  we  have  named  the  Subjective  Workload 
Assessment  Technique  (SWAT). 

To  develop  SWAT  we  defined  workload  as  being  primarily  composed  of 
three  dimensions:  (1)  Time  load,  (2)  Mental  effort  load,  and  (3)  Psycho¬ 
logical  stress  load.  This  multidimensional  definition  of  workload  is 
consistent  with  a  large  body  of  current  workload  research  (Johannsen, 
et  al,  1979;  Senders,  1979;  Williges  and  Wierwille,  1979;  Johns,  1973; 
White,  1971;  Sheridan  and  Simpson,  1979).  In  particular,  this  framework 
and  the  definitions  of  the  three  dimensions  are  adaptations  of  work 
performed  by  Sheridan  and  Simpson  (1979)  in  development  of  a  Cooper-Harper 
type  of  category  scale  for  workload.  Each  of  the  three  dimensions  has 
three  levels  corresponding  roughly  to  high,  medium,  and  low  loading. 
Precise  definitions  and  a  more  in-depth  discussion  of  SWAT  development  can 
be  found  elsewhere  (Reid,  et  al ,  1981a;  1981b). 

These  definitions  of  three  levels  of  each  of  three  dimensions  are 
combined  through  a  mathematical  procedure  known  as  conjoint  measurement. 
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This  portion  of  the  SWAT  procedure  we  have  named  Scale  Development. 

During  scale  development  information  is  obtained  from  subjects  regarding 
the  way  the  three  dimensions  theoretically  combine  to  produce  workload. 

This  information  is  collected  by  having  subjects  order  all  possible 
combinations  of  the  definitions.  This  means  all  possible  combinations 
of  the  three  levels  of  the  three  dimensions  or  27  combinations.  The 
ordering  is  a  separate  distinct  process  which  is  carried  out,  preferably, 
before  starting  an  experiment.  The  data  obtained  is  then  tested  for 
agreement  among  subjects.  If  sufficient  agreement  exists,  say  a 
Kendall's  coefficient  of  condordance  greater  than  .75,  then  the  remainder 
of  the  procedure  is  based  upon  group  data.  It  is  possible,  however,  to 
develop  an  independent  scale  for  each  subject  in  the  event  that  there  is 
low  agreement. 

After  deciding  whether  to  obtain  one  scale  to  represent  the  group  or 
a  scale  for  each  subject  the  data  is  tested  by  a  procedure  called  conjoint 
measurement  (Krantz,  1964;  Krantz  and  Tversky,  1971;  Luce  and  Tukey, 

1964).  This  procedure  tests  the  ordering  data  to  determine  the  combination 
rule  used  by  the  subjects.  Several  rules  or  models  are  possible  (e.g. 
additive,  distributive,  dual  distributive)  but  all  applications  of  SWAT 
have  resulted  in  an  additive  model. 

Once  the  model  is  defined,  the  data  is  transformed  via  an  iterative 
procedure  known  as  conjoint  scaling.  The  transformation  fits  the  data  to 
the  defined  model  and  maintains  the  order  inherent  in  the  orginal  data. 

The  result  is  a  undimensional  interval  level  scale. 
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The  event  scoring  phase  of  SWAT  is  the  process  of  obtaining  workload 
ratings  within  the  context  of  the  investiators  experiment  or  evaluation. 
This  is  done  by  having  subjects  provide  ratings  of  the  event  in  terms  of 
the  levels  of  the  three  dimensions  inherent  in  the  current  situation. 

For  example,  a  subject  might  assign  a  3  for  high  time  load,  a  2  for 
moderate  mental  effort,  and  a  2  for  psychological  stress  as  his  ratings 
for  coping  with  a  particular  in-flight  emergency.  These  ratings  then 
might  correspond  to  a  value  of  125  on  the  previously  developed  scale  and 
this  value  is  the  workload  level  associated  with  the  event. 

The  initial  step  in  the  systematic  development  of  SWAT  was  to  see  if 
subjects  could  perform  the  required  orderings,  and  if  so,  what  level  of 
agreement  exists  between  subjects  and  within  subjects.  The  first 
question  has  been  addressed  by  having  eight  groups  of  subjects  perform 
the  ordering  procedure.  The  entire. procedure  normally  takes  approximately 
forty-five  minutes.  Kendall's  coefficients  of  concordance  associated  with 
these  groups  ranged  from  .79  to  .87  indicating  statistically  reliable 
agreement  within  the  groups.  The  coefficient  of  concordance  obtained 
from  the  pooled  data  of  all  subjects  is  .80  indicating  high  between  group 
agreement. 

Reliability  checks  are  made  for  individual  subjects  by  follow-up 
administration  of  a  set  of  paired  comparisons.  Subjects  are  presented 
with  the  set  of  three  descriptions  describing  two  workload  conditions  and 
are  asked  to  indicate  which  set  represents  highest  workload.  Subjects  are 
presented  nine  of  these  pairs  where  three  represent  very  fine  discrimina¬ 
tions,  three  represent  moderate  discrimination  and  three  represent 
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discrimination  of  situations  clearly  having  different  degrees  of  workload. 
Using  this  technique,  subjects  have  been  found  to  be  approximately  80% 
reliable  when  checked  about  one  month  of  scale  development  and  approximately 
75%  reliable  after  a  lapse  of  about  four  months. 

In  order  to  assess  validity  of  SWAT  an  experiment  was  conducted 
where  subjects  were  presented  with  eight  aircraft  communications  tasks 
while  performing  a  critically  unstable  tracking  task.  The  communications 
tasks  had  previously  been  scaled  using  an  information  theoretic  analysis 
(Shingledecker  et  al,  1980)  so  that  the  eight  task  were  known  a  priori  to 
possess  varying  levels  of  task  demand.  Additionally,  two  levels  of  task 
difficulty  were  use.  Thus,  sixteen  dual  task  conditions  were  presented. 

The  results  of  this  experiment  (Reid,  et  al,  1981a)  demonstrated  that  SWAT 
was  able  to  discriminate  between  the  two  levels  of  primary  task  as  well  as 
between  the  eight  levels  of  the  secondary  task. 

Additional  validation  data  has  been  obtained  through  implementing 
SWAT  as  a  workload  assessment  instrument  in  systems  tests.  The  most 
noteworthy  of  these  applications  is  the  Advanced  Medium  Range  Air-to-Air 
Missile  (AMRAAM)  Operational  Utility  Evaluation  (OUE).  The  OUE  was  a 
large  scale  AIMVAL/ACEVAL  type  of  exercise  conducted  by  the  Air  Force 
Test  and  Evaluation  Center  (AFTEC)  using  the  MAC  AIR  combat  simultor  at 
McDonnell  Douglas  Corporation,  St  Louis,  MO.  In  the  course  of  this 
evaluation  we  collected  over  12,000  event  scores  from  a  pool  of  fifty- 
six  pilots.  A  review  of  the  frequency  distribution  of  these  scorings 
reveals  that  the  whole  range  of  the  twenty-seven  point  scale  is  used. 
Additional  information  from  this  test  showed  workload  differences  in  places 
and  in  directions  that  can  be  logically  defended.  The  exact  findings 
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cannot  be  discussed  here  due  to  the  classification  of  the  data.  How¬ 
ever,  post  test  questionnaires  and  pilot  interviews  were  administered 
and  the  subjective  opinions  of  the  test  participants  support  the  findings 
obtained  from  statistical  analysis  of  the  event  data. 

Additional  applications  of  SWAT  are  currently  in  the  planning  stages. 

An  AFAMRL/AFWAL  program  (RAM/ACE)  to  investigate  potential  cockpit  enhance¬ 
ments  is  employing  a  modified  projective  procedure  using  SWAT.  The  AFTI- 
16  program  as  well  as  the  single-seat  night  attack  study  will  provide  the 
first  flight  test  applications. 

Refinement  of  the  technique  will  continue  as  we  gain  experience  in 
a  wide  variety  of  applications  in  order  to  provide  a  general  workload 
measure.  Data  is  being  collected  regarding  SWATs  sensitivity,  validity 
and  reliability  in  order  to  aid  potential  users  apply  the  technique  to 
their  own  specific  application. 
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Assessment  of  Reliability  and  Validity  for  the 
Systems  Operability  Measurement  Algorithm  (SOMA) 

The  Systems  Operability  Measurement  Algorithm  (SOMA)  (reference 
1)  was  developed  as  a  system  evaluation  methodology  to  provide  decision 
makers  with  information  about  system  adequacy  and  design  alternatives 
within  the  context  of  mission  needs  and  requirements.  During  the  past 
year  a  research  effort  has  been  directed  toward  evaluation  of  SOMA  in 
terms  of  its  reliability  and  validity. 

The  SOMA  model  has  evolved  directly  from  multi -attribute  utility 
(MAU)  theory  (reference  2).  MALI  is  a  Bayesian  oriented  decision  making 
paradigm  (reference  3)  which  processes  information  according  to  a  set 
of  specific  rules. 

There  are  three  rules  of  MAU  which  are  particularly  important  to 
SOMA.  First,  the  basic  structural  principle  in  MAU  is  hierarchical 
decomposition.  This  means  that  the  evaluation  problem  is  broken  down 
from  general  to  specific  components.  The  model  provides  the  structure  and 
rules  necessary  to  investigate  and  integrate  the  interrelationships  of 
all  components.  Second  the  MAU  model  involves  the  definition  of 
utility  (effective  operability).  Optimum  evaluation  of  alternatives  is 
dependent  upon  the  selection  of  a  single  criterion.  Multidimensional 
outcomes  must  be  transformed  into  a  single  figure  of  merit  such  as 
utility  system  worth  or  operational  effectiveness.  Third,  the  MAU 
model  utilizes  the  scaling  of  the  selected  criterion. 
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These  three  aspects  of  MAU  were  addressed  in  the  development  of 
SOMA.  As  part  of  SOMA  development,  task  analysis  methodology  was  used 
to  establish  the  decomposition  hierarchy  and  to  provide  the  basis  for 
the  utilization  of  operability  measures  and  scaling  theory.  These 
scaling  methodologies  are  employed  to  develop  the  rating  scales  for  the 
criticality  (C),  pilot  workload  (PW)  and  technical  effectiveness  (TE) 
dimensions  of  operability,  and  to  provide  the  basis  for  integrating  PW 
and  TE  into  a  single  intervalized  effective  operability  (EO)  dimension. 

As  illustrated  in  Figure  1,  this  approach  for  system  evaluation 
integrates  the  technologies  of  task  analysis,  multi -attribute  utility 
theory,  and  scaling  theory  into  one  functional  algorithm.  SOMA  system¬ 
atically  structures  operator  tasks  (Figure  2)  in  accordance  with  mission 
needs  hierarchy  (Figure  3)  and  then  provides  for  assessment  of  system 
operability  through  application  of  a  linear  conjoint  measurement  scaling 
model  (Figure  4).  All  assessments  are  then  integrated  by  the  rules 
established  through  MAU  theory.  The  output  from  SOMA  is  quantitative 
information  about  the  operability  of  an  entire  weapons  system  (i.e., 
fighter  or  attack  aircraft),  the  operability  of  specific  systems  (such  as 
radar,  communication  or  navigation)  and,  finally,  the  operability  of  each 
task  performed  during  a  mission  phase.  In  short,  SOMA  measures  subsystem 
and/or  system  goal  attainment. 


Insert  Figures  1-4  About  Here 
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In  order  to  demonstrate  the  validity  and  reliability  of  the  overall 
SOMA  technique,  the  reliability  and  validity  of  each  component  part  must 
first  be  demonstrated.  Although  SOMA  is  currently  being  used  as  an 
operational  test  and  evaluation  tool  by  the  U.S.  Air  Force,  it  is 
appropriate  to  examine  the  psychometric  characteristics  of  SOMA. 
Examination  of  these  characteristics  will  provide  the  basis  for  further 
operational  refinement  of  SOMA.  It  is  thus  necessary  to: 

•  Compute  the  test-retest  reliability  (pilot  rating  of  tasks 
over  time)  demonstrated  for  the  criticality  (C),  technical  effectiveness 
(TE),  pilot  workload  (PW)  and  effective  operability  (EO)  scales  which 
compose  the  SOMA  model . 

•  Compute  the  interrater  reliability  (pilot  concurrence  on 
scale  meaning  and  agreement)  for  the  C,  TE,  PW  and  EO  scales  of  the 
SOMA  model . 

•  Compute  internal  consistency  reliability  of  pilot  responses' 
for  the  MAU's  task  inventory. 

•  Assess  the  face  validity  of  SOMA,  i.e.,  determine  whether  users 
agree  that  the  instrument  and  the  scales  adequately  measure  tasks  important 
to  the  mission. 

§  Assess  the  construct  validity  of  SOMA,  i.e.,  empirically 
examine  the  hypothesized  relationships  among  the  structural  components 
of  the  SOMA  model . 
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In  order  to  address  these  reliability  and  validity  questions, 

A-7  and  F-16  pilot  inventory  responses  were  analyzed.  Questions  of 
reliability  are  addressed  by  employing  a  variety  of  standard  reliability 
analyses  and  estimates  of  concordance.  Face  validity  is  addressed  by  a 
recapitulation  of  both  the  development  of  the  hierarchical  task 
decomposition  structure  and  the  rating  scale  structures  along  with  the 
feedback  from  pilots  and  test  evaluators.  Construct  validity  is  addressed 
by  applying  correlational  and  regression  analysis  techniques,  including 
that  of  path  analysis. 
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RESULTS  AND  DISCUSSION 


RELIABILITY 

INTERNAL  CONSISTENCY 

Internal  consistency  analyses  were  performed  on  four  sets  of  data 
which  included  two  groups  of  seven  A-7  pilots  and  two  groups  of  nine 
F-16  pilots.  There  were  167  A-7  and  221  F-16  tasks  analyzed  covering  pre¬ 
launch,  emergency,  and  Visual  Flight  Rules  (VFR)  functions  in  both  tactical 
and  non-tactical  flight  modes.  As  shown  in  Table  1  the  internal  test 
consistency  coefficient,  Cronbach's  alpha  (reference  4)  for  each  data  set 
exceeds  0.95.  This  measure  indicates  the  degree  to  which  the  ratings  across 
tasks  are  similar.  It  appears  that  the  test  items  are  essentially  tapping 
a  single  factor  or  dimension;  that  is,  EO  is  a  unitary  measure.  Furthermore, 
with  the  acceptance  of  an  internal  consistency  index  of  0.80,  the  number  of 
task  elements  required  could  be  reduced  by  approximately  one-half  of  the 
167  items  for  the  A-7  groups.  A  similar  reduction  in  the  size  of  the  F-16 
task  inventory  would  also  be  permitted  with  this  lower  reliability  requirement. 
The  similarities  in  consistency  measures  for  the  A-7  and  F-16  data  sets 
provide  evidence  that  the  SOMA  methodology  has  produced  inventories  and 
structures  that  are  equivalent  in  terms  of  internal  structure  and  factor 
composition. 


Insert  Table  1  Here 


294 


INTERRATER  RELIABILITY 


Analyses  examining  the  agreement  among  raters  were  performed  on 
operability  scores  for  the  A-7  and  F-16  data.  Figure  5  summarizes  the 
interrater  reliability  plotted  as  a  function  of  subject  sample  size. 

For  the  sample  range  of  four  to  seven  (A-7)  and  four  to  nine  (F-16), 
subjects  were  randomly  dropped  from  each  data  set  before  reliabilities 
were  recalculated.  The  reliability  of  EO  (0.60)  appears  acceptable  at 
and  above  a  sample  size  of  six.  Reliability  drops  more  rapidly  as  N  is 
reduced  below  six  observers.  This  finding  supports  the  notion  that  factors 
such  as  training  level  and  perceptual  set  for  each  rating  scale  are 
sufficiently  controlled  if  N  is  equal  to  or  greater  than  six. 


Insert  Figure  5  Here 


Figure  6  presents  similar  ascending  reliability  coefficients  for 
the  two  F-16  data  sets  as  N  is  increased  from  4  to  9  for  the  C  and  the 
TE  scales.  On  the  other  hand,  PW  reliability  indices  are  relatively 
low  (ranging  from  0.25  to  0.58)  and  vary  considerably  between  test  groups. 
The  test-retest  reliability  for  a  subset  of  six  pilots  who  performed  both 
tests  was  very  high  (0.96).  That  is,  although  individual  pilots  retained 
stable  workload  criteria  (test-retest),  there  is  substantial  variation  in 
criteria  between  pilots.  If  this  is  the  case,  then  either  more  effort  must 
be  made  to  normalize  pilot  workload  criteria  or,  as  an  alternative,  the 
maximum  number  of  pilots  should  be  employed  to  extract  an  acceptable  level 
of  interra'ter  reliability  on  pilot  workload  measures. 
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Insert  Figure  6  Here 


INTERRATER  CONCORDANCE 

Concordance  analyses  (reference  6)  were  performed  for  the  A-7  and 
F-16.  As  shown  in  Table  2,  concordance  remains  above  0.90.  These 
results  confirm  the  hypothesis  that  an  averaged  matrix  can  be  employed  to 
generate  the  operability  matrix  for  each  data  set.  In  addition,  the  overall 
concordance  analysis  in  which  matrix  ratings  for  F-16  and  A-7  groups  were 
combined  indicates  a  concordance  ratio  (Kendall’s  W)  of  0.96.  This 
result  indicates  that  the  scale  structure  for  PW,  TE,  and  E0  are  common 
across  pilot  groups.  Thus,  a  standard  matrix  for  operability  can  be 
constructed  which  is  suitable  for  all  of  the  E0  tasks  involving  single 
seat  fighter  and  attack  aircraft. 


•  Insert  Table  2  Here 


INTRA-RATER  RELIABILITY 

Two  groups  of  F-16  data  were  analyzed  for  test-retest  reliability 
(reference  5)  for  189  of  the  221  items  in  the  completed  inventory  (Table  3). 
The  32  pre-flight  task  items  were  excluded  from  the  inventory  due  to  computer 
software  limitations.  These  items  were  assumed  to  be  the  most  reliable  of 
the  task  set.  The  calculated  reliability  estimates  are  thus  conservative. 
Note  that  in  Table  3  the  six  pilots'  test-retest  reliability  for  E0  was  0.72. 
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That  is,  over  50%  of  the  variance  was  accounted  for  by  these  tests  in 
achieving  the  EO  scores.  With  the  assumption  that  no  additional  training 
or  experience  was  obtained  between  tests,  the  resultant  reliability 
coefficient  would  still  be  viewed  as  an  acceptable  test-retest  level. 
However,  all  of  the  pilots  had  additional  experience  with  the  aircraft 
system.  This  added  flight  time  probably  alters  pilot  evaluation  criteria, 
thus,  decreasing  reliability.  The  achieved  test-retest  coefficient  is 
thus  viewed  as  a  conservative  estimate. 


Insert  Table  3  Here 


TEST-RETEST  RELIABILITY  FOR  C,  PW,  TE 

Reliability  coefficients  were  forced  for  the  ordinal  C,  PW  and  TE 
scale  results  over  the  F-16  189  item  inventory.  The  TE  and  PW  scale  scores 
(Table  3)  within  pilots  remained  stable  across  tests  (reliability  coefficients 
of  0.87  and  0.96,  respectively).  The  C  ratings,  on  the  other  hand,  were 
less  stable  (reliability  coefficient  =  0.63).  The  above  findings  contrast 
with  the  interrater  reliability  indices  obtained  on  these  scales.  PW 
varied  between  pilots  and  remained  stable  within  pilots  while  C  ratings  tended 
to  be  similar  between  pilots  but  varied  within  pilots.  These  contrasts 
provide  evidence  that  external  factors  such  as  examiner  input  or  biases 
influence  scales  such  as  C.  It  is  also  apparent  that  pilots  tend  to  form  a 
constant  concept  of  workload,  but  each  pilot  maintains  his  own  set  of 
criteria  for  this  scale  dimension. 
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TEST-RETEST  MATRICES 


Operability  matrices  constructed  by  the  six  F-16  pilots  who  participated 
in  both  tests  were  analyzed  for  test-retest  reliability  by  forcing  the  data 
with  the  standard  reliability  analysis.  The  obtained  reliability 
coefficient  (0.99)  further  strengthens  the  argument  that  a  standard 
rating  matrix  is  achievable  for  fighter  and  for  attack  single  seat  aircraft 
SOMA  analyses. 


VALIDITY 


FACE  VALIDITY 


Face  validity  is  extremely  important  to  any  evaluation  tool  which 
depends  on  user  input.  The  iterative  MAU  process  used  to  develop  the 
SOMA  model  incorporated  operational  pilots  into  the  process  throughout 
the  development  of  the  model  for  each  airframe.  The  following  is  a 
description  of  the  hierarchical  decomposition  development  process  and  the 
scale  generating  process-  that  illustrates  user  integration  into  these 
processes.  Mission  requirements  and  pilot  functions  were  task  analyzed; 
a  preliminary  hierarchical  decomposition  was  then  developed.  Pilots  were 
then  requested  to  evaluate  the  structure.  The  pilots’  feedback  on  the 
fidelity  and  utility  of  the  hierarchy  served  as  the  basis  for  any  re¬ 
structuring  of  the  hierarchy.  The  second  iteration  was  also  submitted 
for  pilot  review.  This  process  continued  until  pilots  were  satisfied 
that  the  decomposable  hierarchy  accurately  reflected  their  mission  require¬ 
ments  and  that  the  structure  flowed  in  correct  mission  sequence. 
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Operational  pilots  were  also  used  in  the  SOMA  process  to  generate 
the  C,  PW  and  TE  rating  scales.  These  scales  were  evolved  through  a 
process  similar  to  the  development  of  task  hierarchies.  Scales  and 
definitions  were  designed  to  cover  the  range  of  operability  along  the 
three  dimensions.  The  definitions  were  structured  to  reflect  mission 
(C),  system  (TE)  and  man  (PW)  dimensions  and  at  the  same  time  maximize 
the  orthogonality  among  the  scales.  A  four  interval  forced  choice 
scale  was  selected  to  sample  more  adequately  the  full  range  of  the  scale. 
The  scales  were  then  evaluated  by  pilots  to  ensure  that  the  scales  were 
useable,  meaningful  and  reflected  the  total  operability  function.  High 
test-retest,  internal  consistency,  and  interrater  agreement  on  each  of  the 
scales  as  well  as  the  output  measure,  EO,  suggest  that  the  scales  and  the 
hierarchies  are  understood  and  useable  by  the  pilots. 

CONSTRUCT  VALIDITY 

The  statistical  procedure  of  path  analysis  (reference  7)  was  used  to 
evaluate  the  construct  validity  of  the  SOMA  model.  Two  path  analyses 
were  calculated  for  the  F-16  data  sets.  The  separate  tests  were  performed 
to  compare  the  results  for  consistency  and  stability  of  both  path 
coefficients.  Table  4  presents  the  data  developed  from  multiple 
regression  analyses  which  provided  the  input  data  for  the  path  analysis 
model  illustrated  in  Figure  7.  The  simple  partial  correlations  (r..), 

I  J 

which  represent  the  total  covariance  on  any  path,  are  taken  when  C,  PW 
and  TE  are  regressed  on  EO.  The  direct  path  coefficients  (P - . )  are 

I  J 

standardized  betas  with  the  lone  exception  being  X«  X.  where  r 

4  34 

and  P34  are  equal  using  the  model  presented  in  Figure  7.  The  indirect 
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causal  effects  are  formed  through  combination  rules  dependent  on  the 
paths  specified  by  the  model.  Total  causal  effects  are  simply  the 
algebriac  summation  of  indirect  and  direct  effects.  For  instance,  in  X2  X^, 
total  causal  variance  =  0.22  +  (-0.12).  Other  factors  which  contribute 
to  the  simple  partial  correlations  (r..)  which  are  considered  non-causal  in 

*  J 

terms  of  the  specified  model  are  derived  by  algebriac  subtraction  of  the 
total  covariation  from  total  causal  covariation.  The  model  represented  in 
Figure  7  was  constructed  by  hypothesizing  that  PW,  TE  and  C  formed  the  basis 
for  the  EO  dimension  used  in  development  of  the  EO  scale  (Figure  4).  The 
C  dimension  is  not  included  in  the  PW  by  TE  matrix.  However,  a  direct 
path  from  C  to  EO  was  hypothesized  to  obtain  a  complete  path  analysis  model. 
The  primary  input  from  C  to  EO  is  assumed  to  occur  through  PW  and  TE.  TE  is 
assumed  to  be  causally  linked  to  PW  because  the  equipment  in  the  system 
is  fixed  and  should  form  one  anchor  in  the  development  of  the  PW  rating. 


Insert  Table  4  Here 


Insert  Figure  7  Here 


Findings  which  confirm  the  validity  of  the  hypothesized  MAU  model 
would  meet  the  following  four  requirements: 

1.  EO  is  entirely  determined  by  the  three  input  dimensions 

2.  Most,  if  not  all,  of  the  PW,  C  and  TE  variability  is  externally 
determined 
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3.  C  demonstrates  no  direct  causality  with  EO  but  demonstrates 
substantial  indirect  causal  covariance 

4.  PW  and  TE  are  independent. 

The  results  of  the  path  analysis  appear  to  satisfy  the  requirements 

for  the  hypothesized  MAU.  For  requirement  1  most  of  the  variance  in 

2 

effective  operability  is  internally  determined  (Group  1,  R  =  0.80; 

2 

Group  2,  R  =  0.94).  For  requirement  2  the  variation  from  external  sources 
on  PW  and  TE  range  from  a  low  of  0.56  for  PW  in  group  2  to  a  high  of  0.89 
for  TE  in  Group  1.  For  requirement  3,  C  shows  no  direct  relation  with 
EO  (Direct  Causal  P14  =  °*  03  and  -0.04)  while  the  indirect  effects  of  C 
on  EO  are  -0.22  and  0.28  for  Groups  1  and  2,  respectively.  For  requirement 
4  the  relatively  strong  relation  (0.32  and  0.68)  between  PW  and  TE,  suggests 
that  these  scales  need  to  be  improved  if  the  orthogonality  assumption  is  to 
be  strictly  satisfied  for  applying  the  conjoint  measurement  approach. 

One  area  where  the  PW  scale  might  be  improved  is  by  making  its  scale 
units  match  those  used  for  TE.  There  is  a  negative  relation  between  C  and 
TE  (-0.2  and  -0.35)  while  the  C  and  PW  relationship  tends  to  be  positive 
(0.22  and  0.06).  This  inverse  relation  between  C  and  TE  and  between  C  and 
PW  tends  to  explain  why  C  relation  has  been  difficult  to  incorporate 
previously  into  the  model.  Inspection  of  the  PW  and  TE  scales  reveals  that 
TE  is  a  positive  scale  while  PW  is  a  negative  scale.  If  both  were  positive 
the  covariation  of  PW  and  TE  with  C  would  be  in  the  same  direction. 
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SUMMARY  AND  CONCLUSIONS 


Reliability  and  validity  analyses  were  performed  on  A-7  and  F-16  data 
subjected  to  the  SOMA  technique.  The  following  conclusions  were  reached: 

t  Task  inventories  for  SOMA  could  be  reduced  by  1/2  without 
excessively  decreasing  internal  consistency  reliability. 

•  Effective  operability  is  essentially  a  single  dimension  or  single 
factor  measure. 

•  SOMA  methodology  has  produced  inventories  and  structures  that  are 
equivalent  in  terms  of  internal  structure  and  factor  composition. 

•  Interrater  reliability  for  EO,  C  and  TE  are  acceptable  when  N  =  6 
or  greater. 

•  Average  rater  matrices  may  be  used  to  develop  the  EO  scale. 

•  A  standard  matrix  for  EO  can  be  constructed  which  is  suitable 
for  single  seat  fighter  or  attack  aircraft. 

§  The  EO  test-retest  ratio  of  0.72  (accounting  for  over  50%  of  total 
variance)  is  a  conservative  reliability  estimate. 

•  Workload  estimates  are  consistent  within  pilots  but  variable 
between  pilots. 

•  Reliability  estimates  verify  that  the  method  of  construction  of 
the  task  hierarchy  and  scales  demonstrate  face  validity. 

§  The  SOMA  model  is  essentially  valid. 

•  The  PW  scale  should  be  adjusted  through  definitions  to 

a.  improve  orthogonality  with  the  TE  dimension 

b.  scale  along  a  monotonic  dimension 
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FIGURE  CAPTIONS 


1.  SOMA  Developmental  Model 

2.  Mission  Task  Hierarchy 

3.  Mission  Needs  Hierarchy  Decomposition  Structure 

4.  Conjoint  Measurement  Methodology 

5.  Effective  Operability  Reliability  by  Number  of  Raters 
for  A-7  and  F-16 

6.  Criticality,  Pilot  Workload,  and  Technical  Effectiveness 
Reliability  by  Number  of  Raters  for  A-7 

7.  SOMA  Path  Model 
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CRITICALITY'  P/U>T  WORKLOAD  TECHNICAL  EFFECTIVENESS 
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number  of  pilots 


SOMA  PATH  MODEL 
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JEXTERNAL 


TABLE  CAPTIONS 


1.  Summary  of  Internal  Consistency  Reliability  for  A-7  and  F-16 

2.  Operability  Matrix  Inter-Rater  Concordance  Ratios  (Kendall's  W) 

3.  F-16  Test-Retest  Reliability  Summary 

4.  Path  Analysis  Summary  Tables 
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TABLE  1.  SUMMARY  OF  INTERNAL  CONSISTENCY  RELIABILITY 
FOR  A-7  AND  F-16 


CO 

o 


cn  cn 


col 


CD  CO 
CO  CD 


CD  CD 
OO  OO 


c_>| 


*  * 


cn  oo 

CD  CD 


Q_l 


*  * 


cn  cn 
cn  cn 


*  * 


cn  cn 
cn  cn 


Si 


CD  OO 

cn  cn 


co  cn 
cn  cn 


o_ 

ZD 

o 

CeZ 

CD 


I  I 

r\  rv. 
<c  <c 


LU 

I— 

Q 

Z 

< 

CD 


Q_ 

lie 


314 


TABLE  2 

OPERABILITY  MATRIX  INTER-RATER 
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From  Helm  and  Donnell,  1979 


TABLE  3, 

F-16  TEST-RETEST  RELIABILITY  SUMMARY 
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MEDIANS 

SEMI-INTERQUARTILE  RANGE 


TABLE  4.  PATH  ANALYSIS  SUMMARY 
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A  SELF-CONTAINED,  MAN-BORNE  BIOMEDICAL  INSTRUMENTATION 
SYSTEM  IN  THE  FLIGHT  TESTING  OF  NAVAL  WEAPONS  SYSTEMS 
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ABSTRACT 

The  historical  lack  of  aircrew  physiological 
data  has  prevented  its  use  in  the  definition  and 
evaluation  of  airborne  man-machine  systems.  Recog¬ 
nizing  this  deficiency,  a  joint  Navy  and  Air  Force 
program  has  resulted  in  the  development  of  a  self- 
contained  man-borne  In-Flight  Physiological  Data 
Acquisition  System  (IFPDAS).  This  system  is  cap¬ 
able  of  obtaining  and  recording  up  to  32  channels 
of  physiological  and  engineering  parameters  for 
flight  durations  to  four  hours.  A  ground  based 
microprocessor  provides  computer  analysis  of  this 
data.  At  present,  the  Navy  is  establishing  a 
methodology  for  the  employment  of  the  system, 
developing  analytical  software,  and  collecting 
operational  flight  data.  Further  plans  for  the 
IFPDAS  include  collecting  and  processing  multi¬ 
parameter  physiological  and  environmental  data  from 
aircrewmen  to  more  thoroughly  describe  the  tactical 
flight  environment  and  thus  support  proper  develop¬ 
ment  of  aircrew  life  support  and  weapons  systems. 

THE  PROBLEM 

Currently  operational  aircraft  such  as  the 
F-14  are  capable  of  performance  which  in  many 
flight  regimes  exceeds  the  pilot's  ability  to  sus¬ 
tain  normal  physiological  function.  The  F/A-18 
undergoing  test  and  evaluation  at  the  Pacific 
Missile  Test  Center  ( PACMISTESTCEN)  presents  a 
still  greater  operator  challenge.  Even  with 
sophisticated  life  support  systems,  man  is  often 
the  limiting  factor  in  the  optimal  employment  of  an 
aircraft  performance  capabilities..-  From  a  weapons 
system  standpoint,  the  aircrewnan's  psychological 
abilities  also  present  a  significant  constraint. 
Man's  limited  sensor  modality,  signal  processing 
capability,  and  pattern  recognition  techniques  are 
no  match  for  today's  accelerating  technology. 

While  considerable  research,,  analysis,  and 
testing  is  conducted  as  part  of  aircraft  and  weapon 
systems  development  to  determine  the  equipment's 
response  to  the  environmental  and  operational 
demands,  relatively  little  is  done  to  characterize 
tne  operator's  psycho-physiological  responses.  The 
design  of  a  crew  station  in  an  airborne  system  is 
typically  an  extrapolation  from  an  existing  system 
tempered  with  subjective  evaluations  from  a  small 
sample  of  system  operators.  With  existing  tech¬ 
niques  it  is  not  currently  possible  to  determine 
in  an  objective  manner,  to  what  extent  the  "human 
factor"  limits  overall  svste~s  performance . 


*  Head,  Crew  Systems  Branch 

+  Naval  Reserve  Aviator 

++  Physical  Scientist,  Crew  Systems  Branch 


This  inability  to  measure  the  operator's 
response  to  the  tactical  flight  environment 
presents  an  additional  problem  to  the  system 
designer.  The  standards  and  specifications  which 
control  the  development  of  life  support,  controls 
and  display  systems  provide  only  limited  guidance 
with  respect  to  operator  workload  and  boundaries 
for  the  operator's  physiological  response.  This 
situation  hampers  the  optimization  of  the  man- 
machine  interface  and,  thereby,  limits  the  perfor¬ 
mance  of  future  airborne  systems. 

Hi  story 

A  considerable  amount  of  in-flight  monitoring 
of  physiological  parameters  was  conducted  by  NASA 
throughout  the  1 960 ' s .  Most  of  this  data  was 
collected  during  special  test  flights  and  not  under 
typical  tactical  conditions.  In  1965,  however, 

NASA  researchers  under  two  separate  projects 
instrumented  Navy^p^lots  involved  in  flight  opera¬ 
tions  in  Vietnam  ’  While  the  aggregate  of  these 
two  projects  was  less  than  fifty  flights  the 
results  unexpectedly  indicated  that  from  a  physio¬ 
logical  standpoint,  carrier  launch  and  recovery 
operations  were  more  disruptive  to  the  pilots  than 
were  combat  operations.  Observations  such  as  these 
and  the  requirement  for  improved  human  engineering 
design  standards  and  specifications  prompted  an 
increasing  interest  in  in-flight  physiological 
monitoring. 

A  U.S.  Navy  developed  man-mounted  physiology 
data  recorder,  the  Bio-Pack  was  used  in  the  late  ^ 
1 960 ' s .  This  unit,  as  described  by  Horrigan  et.al 
employed  a  small  8  channel  reel-to-reel  tape 
recorder  to  store  up  to  40  minutes  of  multiplexed 
data.  Parameters  obtained  were  electrocardiograph 
(ECG),  body  temperature  and  voice  from  the  aircrew- 
man  plus  cockpit  accelerations  and  temperatures. 

The  Bio-Pack  recorder  with  its  battery  power  source 
was  packaged  in  a  6"  x  2"  x  10"  container  and 
weighed  £.7  pounds.  It  could  be  placed  conven¬ 
iently  on  the  pilots  knee  boara  or  carried  in  the 
map  case  or  any  adjacent  cockpit  area.  Over  30 
successful  Bio-Pack  instrumented  flights  were  con¬ 
ducted.  Data  from  these  tests  exhibited  close 
correlations  between  the  stressful  events  of  the 
flights  and  the  aircrew's  physiological  responses. 
The  investigators  were  pleased  with,  the  Bio-Pack's 
performance  but  reported  plans  to  extend  the  40 
-.inute  recording  capability  ar.d  to  develop  a 
smaller  lighter  package.  Suer  a  unit  woulc  more 
closely  meet  tne  design  objectives  cf  non-inter¬ 
ference  with  the  aircrewman,  compactness,  reli¬ 
ability,  and  versatility.  The  U.S.  Air  Force  at 
Brooks  Air  Force  Base  attempted  to  meet  these 
criteria  in  1975  with  a  physiologic  recorder 
purenased  on  contract  from  SOI  Systems  Inc.  o f 
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Houston,  Texas.  This  recorder,  termed  IFPDAS, 
provided  for  continuous  simultaneous  recording  on 
cassette  tape  of  seven  channels  of  physiological 
and  environmental  data.  These  include  ECG, 
expiratory  flow,  pO-  (inhaled/exhaled  Oj  concen¬ 
tration),  G, ,  acceleration  (vertical  to  the  pilot), 
caoin  oressure,  voice  information^and  a  time  code, 
"he  system  as  described  by  Morgan  was  inrenced  to 
oetermine  breathing  system  design  parameters  and 
investigate  other  biomedical  requirements  of 
flight.  Navy  aircrewmen  at  the  PACMISTESTCEN  were 
monitored  with  the  IFPDAS  unit  while  flying  A-4, 

F-4  and  F-14  aircraft.  This  data  along  with  meas¬ 
urements  obtained  from  Air  Force  C-130  &  T-3-$  . 
pilots  were  reported  in  1  975  ,  1  976  and  1977. 

During  these  flights,  the  components  of  the  IFPDAS 
were  consolidated  into  pockets  on  the  pilot's 
survival  vest.  This  eliminated  the  complicated 
umbilical  connections  required  to  accomodate 
possible  ejection  and  man-seat  separation  contin¬ 
gencies.  Thus  air-crewmen  began  to  more  readily 
accept  the  slight  encumbrance  produced  by  the 
recorder  components  and  became  more  willing  to  wear 
the  IFPDAS  on  any  mission. 

One  of  the  most  valuable  applications  of 
IFPDAS  was  to  measure  breathing  requirements  from 
pilots  of  several  different  tactical  aircraft 
under  various  flight  conditions  to  determine  if 
current  oxygen  delivery  standards  were  realistic. 
One  study6  noted  that  expiratory  flow  rates  from 
pilots  in  T-38,  A-4,  F-4  &  F-14  aircraft  typically 
ranged  from  12.5-23.1  liter/min.  on  take  off  and  as 
high  as  28.2  liters/min.  while  landing.  Of  major 
significance  was  the  finding  that  measured  . 
increases  in  minute  ventilation  often  exceeded 
levels  commonly  Referenced  in  breathing  system 
design  parameters'.  Thus  there  are  times  during 
stressful  flight  events  when  the  current  USAF  0, 
delivery  standard  of  25.1  liters/min  and  the  USN 
standard  of  28.0  liters/min  are  exceeded.  These 
standard  values  were  usually  obtained  from  alti¬ 
tude  chamber  situations.  Morgan  et.al5  recom¬ 
mended  that  future  standards  require  oxygen  systems 
to  deliver  up  to  60  liters/min  during  transient 
periods  of  extreme  need,  in  addition  to  the 
routinely  encountered  levels  of  20-25  liters/min 
during  normal  flight.  Morgan  further  suggested 
that  when  new  missions  or  aircraft  were  considered 
in-flight  pulmonary  monitoring  should  be  conducted 
to  assure  that  the  respiratory  life  support  systems 
be  truly  able  to  support  the  aircrew. 

In  1976,  the  USAFSAM  repackaged  and  integrated 
the  IFPDAS  for  flight  in  Naval  aircraft.  The  new 
recorder,  termed  IFPDAS  II,  was  flown  on  approx¬ 
imately  30  flights  at  the  PACMISTESTCEN  between 
1976  and  1979  in  T-2,  TA-4,  A-7,  F-4  and  F-14 
aircraft  in  a  wide  variety  of  flight  missions 
Including  missile  runs,  strafing  and  weapon  re¬ 
lease.  Functional  capability  of  the  IFP0AS  was 
demonstrated  on  flights  to  airspeeds  of  mach  1.5 
and  during  maneuvers  producing  up  to  +5  Gz  .  While 
these  tests  were  underway  the  USAF  and  the  USN 
made  plans  to  develop  an  entirely  new  IFPDAS  with 
even  greater  capabilities. 

In  1977,  the  PACMISTESTCEN  Microelectronics 
Laboratory  and  Weapons  Instrumentation  Division 
began  work  on  the  third  generation  IFPDAS.  Previous 
programs  had  shown  that  the  collection  of  high 
quality  data  was  only  one  part  of  the  physiological 
assessment  process.  Manual  techniques  for  tabula¬ 


tion  and  analysis  of  these  large  quantaties  of  raw 
data  were  unacceptable.  In  addition  to  the  sensors 
and  recorder  package  common  to  all  previous  sys¬ 
tems,  the  goal  of  the  IFPDAS  III  development  pro¬ 
gram  was  to  provide  a  data  analysis  capability. 

The  heart  of  this  system  was  to  be  a  small  mini 
ccmouter  which  could  employ  software  controlled 
pattern  recognition  and  statistical  analysis 
routines  to  large  volumes  of  flight  data. 

IFPDAS  III  was  completed  in  1979.  It  then 
entered  an  extensive  test  and  evaluation  process 
directed  at  validation  of  the  design  and  environ¬ 
mental  qualification  (calibration  of  all  the  system 
ccroonents ; .  Preliminary  efforts  were  also  mada  to 
develop  basic  software  for  the  system. 

IFPDAS  III 

IFPDAS  III  components  are  identified  in  a 
functional  flow  diagram  (Figure  1).  The  man 
mounted  self-contained  recorder  weighs, just  3  1/2 
lbs.  and  its  small  size  (6"  x  5"  x  2")  allows  it  to 
fit  comfortably  in  the  aircrewnan 1  s  survival  vest. 
Additional  man-carried  components  of  IFPDAS  III 
include  a  pulmonary  flow  unit  and  a  cardiothermal 
module. 

The  airborne  recorder  is  capable  of  monitoring, 
conditioning  and  recording  up  to  20  channels  of 
data  for  up  to  four  hours;  or,  when  combined  with 
the  cardio-thermal  module,  it  can  record  a  mix  of 
12  channels  analog  and  32  channels  digital  data. 

The  data  are  multiplexed  on  a  four  track  magnetic 
tape  cassette.  The  specific  data  currently  avail¬ 
able  on  the  Navy  IFPDAS  III  include  expired 
respiratory  gas  flow  rates,  ECG/heart  rate,  skin 
temperatures,  acceleration,  cabin  pressure,  voice 
annotation  and  a  timing  signal  . 

ECG/Heart_Rate:  Both  ECG  and  a  digital  record  of 
heart  rate  are  recorded  from  a  3  lead  chest  mounted 
electrode  harness. 

Expiratory  Flow  Rate:  A  sensor  in  the  expiration 
port  of  the  Navy  A-13A  oxygen  mask  sends  a  pneu¬ 
matic  signal  to  a  flow  transducer  in  the  pulmonary 
flow  module  which  in  turn  produces  an  analog 
electrical  output. 

Skin/Cockpit  Temperatures:  Up  to  8  skin  and- 2 
cockpit  thermistors  provide  inputs  to  the  cardio¬ 
thermal  module.  Output  to  the  recorder  is  a 
digital  signal. 

Voice:  All  communications  are  monitored  and 
recorded  by  means  of  a  sidetone  tap  into  the 
headset  side  of  the  mask  interphone  cord. 

-Gz  Accelerations:  ^Gz  accelerometer  (-3  to  +10  Gz 
range)  is  located  within  the  recorder  package. 

Cockpit  Pressure:  An  aneroid  barometer  in  the 
recorder  module  measures  ambient  cockpit  pressure 
(0-760  mm  Hg) . 

Timing:  An  internal  clock  generates  a  16  BIT 
timing  code  to  correlate  all  measured  parameters. 

The  recorder  can  monitor,  condition  and  re¬ 
cord  data  for  4  continuous  hours.  The  conditioned 
signals  from  the  sensors  are  time  multiplexed  by 
three  8  channel  analog  multiplexers  and  recorded 
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on  a  standard  4  track  magnetic  tape  cassette  using 
a  pulse  duration  modulation  (PDM)  format.  The  4th 
track  is  used  for  sidetone  voice  recording. 

The  ground  based  components  of  the  system 
include  a  data  reproducer  playback  device,  a 
portable  field  data  processor,  teleprinter  and 
laboratory  data  processor.  For  data  analysis  the 
IFPDAS  reproducer  converts  the  PDM  signals  from  the 
recorder  to  analog  signals.  The  voice  channel  is 
fed  into  an  audio  amplifier  and  speaker  within  the 
unit.  Using  the  reproducer’s  control  options,  any 
of  the  analog  signals  can  be: 

a.  Displayed  as  a  voltage  on  a  liquid 
crystal  display  in  the  reproducer. 

b.  Fed  into  a  strip-chart  recorder 

c.  Output  to  the  field  data  processor. 

The  reproducer  is  capable  of  being  operated  in 
a  pre-programmed  time  mode  in  which  the  start  and 
stop  times  of  the  required  portion  of  the  data  may 
be  entered  via  the  keyboard  and  an  automatic  tape 


search  is  initiated  and  controlled  by  the  processor 

In  the  field  data  procesor,  analog  data  from 
the  reproducer  is  converted  to  digital  data  by  an 
A  to  D  converter.'  A  control  tape  in  the  processor 
controls  the  output  of  the  reproducer;  i.e., 
rewind,  fast  forward,  play.  The  microprocessor  In 
the  field  data  processor  was  designed  with  integral 
analog  multiplexers  (32  channels),  a  10  bit  analog- 
to-digital  converter,  and  digital  input/output 
ports.  The  device  uses  11  kilobytes  of  erasable- 
programmable  read-only  memory  and  24  kilobytes  of 
random-access  memory.  The  field  data  processor 
allows  manipulation  of  the  data  using  test  oriented 
BASIC  language.  Processed  data  is  displayed  on  a 
teleprinter.  The  data  can  be  printed  in  either 
tabular  or  plot  formats.  The  IFPDAS  recorder  does 
not  require  aircraft  modification  or  installation; 
no  aircraft  resources  are  required,  no  te'^ametry 
channels  or  aircraft  operational  priorities  are 
necessary  and  the  system  is  not  aircraft-type 
specific.  Therefore,  all  support  features  common 
to  usual  test  installations  are  eliminated. 


USN  IFPDAS  FUNCTIONAL  FLOW  DIAGRAM 
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SYSTEM  APPLICATION 

Concurrent  with  the  development/val idation 
process,  the  IFPDAS  III  unit  has  been  used  to 
collect  and  analyze  data  for  several  ongoing  U.S. 
Air  Force  and  Navy  programs.  Following  is  a  brief 
description  of  these  projects  and  the  role  of  the 
IFPDAS. 

USAF  Cockpit/cabin  temperature  data,  aircrew  heart 
rates  in  C- 130,  C-5A,  C- 141,  F-15  aircraft. 
Environmental  control  systems  evaluation,  aircrew 
heart  rate  and  skin  temperatures,  A-7K  aircraft. 

Navy  Crew  station  accelerations  and  crew  heart 
rate  responses  on  SEPTAR  Target  boats.  General 
aircrew  workload  in  A-7,  A-6  and  F-14  aircraft. 

Current  Plan  During  the  ongoing  engineering 
development  phase  of  this  program,  IFPDAS  capa¬ 
bilities  will  be  expanded  in  three  specific  areas. 
These  include: 

a.  Gaining  experience  in  collection  of 
laboratory  and  in-flight  data  to  verify  and  refine 
data  acquisition  methodology; 

b.  Development  and  verification  of  software 
routines  to  allow  more  sophisticated  data  analysis; 

c.  Gaining  operational  experience  by  deploy¬ 
ing  the  IFPDAS  to  a  remote  operating  location 
(deployed  unit)  or  aboard  an  aircraft  carrier  for 
extensive  use  away  from  readily  available  main¬ 
tenance  facilities. 

This  phase,  will  conclude  with  the  formal  document¬ 
ation  of  the  procedural  and  analytical  methodology 
for  the  effective  employment  of  this  testing  tool. 
Then  the  IFPDAS  can  be  used  to  help  answer  such 
vital  questions  as: 

1.  How  is  the  aircrevanan's  efficiency  affect-, 
ed  by  present  heavy,  bulky,  personal  protective 
equipment  and  arelife  support  systems  really 
supportive? 

2.  Do  new  weapons  systems  platforms  exceed 
the  physiological  capabilities  of  the  aircrewmen? 

3.  Can  information  on  operator  workload  be 
used  to  develop  more  efficient  training  procedures 
and  simulators? 
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Physiological  and  Performance  Parameters  as 


Indices  of  Pilot  Workload 


An  Analysis  of  Data  from  the  AFTI  F-16  Pro.ject 


by 

Wg.  Cdr.  D.  C.  Reader,  RAF 
USAF  School  of  Aerospace  Medicine 
Brooks  AFB,  Texas  78235 


INTRODUCTION 

One  important  new  project  in  fighter  aircraft  technology  is  the  Advanced 
Fighter  Technology  Integration  (AFTI)  F-16.  The  performance  of  this  airplane 
will  be  enhanced  by  the  installation  of  aerodynamic  canards  to  the  underside 
of  the  engine  air  intake.  Deflection  of  these  surfaces  together  with  the  rud¬ 
der,  elevators,  and  flaps  will  allow  the  aircraft  to  translate  vertically  and 
laterally  or  hold  constant  pitch  or  yaw  attitudes.  This  will  reduce  the 
vulnerability  of  the  aircraft  without  affecting  its  firepower.  These  computer- 
controlled  aerodynamic  surfaces  will  greatly  increase  the  performance  of  the 
airplane.  Thus  the  limitation  to  the  weapon  system  may  well  be  the  pilot;  by 
his  inability  to  sustain  high  accelerations  in  the  vertical  and  lateral  axes 
and  his  mental  processing  time  which  could  be  considerably  longer  than  that  of 
the  aircraft  computers. 

In  order  to  reduce,  as  much  as  possible,  this  limitation  to  the  overall 
weapon  system,  the  USAF  School  of  Aerospace  Medicine  was  asked  to  devise  meth¬ 
ods  to  measure  the  workload  of  the  pilot  to  predict  when  limitations  would 
occur  and  to  advise  what  improvements  were  necessary  to  enhance  the  perform¬ 
ance  of  the  pilots. 

Initially,  much  of  the  development  work  would  be  performed  in  simulators; 
a  fixed  base  simulator  (FBS)  at  General  Dynamics,  Inc.,  at  Fort  Worth,  Texas, 
and  the  Dynamic  Environmental  Simulator  (DES)  at  the  Air  Force  Aerospace  Med¬ 
ical  Research  Laboratory  at  Wright-Patterson  AFB,  Dayton,  Ohio.  During  this 
simulation  effort,  physiological  and  performance  data  were  recorded  from  the 
pilots  and  analyzed  at  the  USAF  School  of  Aerospace  Medicine.  This  paper  de¬ 
scribes  the  data,  the  analyses  and  the  conclusions  drawn. 

METHODS 

The  physiological  data  recorded  were  the  electrocardiogram  (EKG)  and 
electromyograms  (EMG)  from  the  right  arm  and  left  neck  areas.  Raw  tracking 
error  data  from  the  simulator  computers  were  also  recorded  as  direct  measures 
of  performance.  These  data  were  amplified,  filtered  and  recorded  on  an  FM 
multi-channel  tape  recorder. 

These  data  were  then  digitized,  summarized  and  subjected  to  analysis  of 
variance.  Throughout  the  simulations  hand  written  logs  in  IRIG  B  Code  were 
made  of  pilot  activity  and  the  tasks  performed  by  each.  The  logs  were  then 
used  to  compare  the  data  throughout  the  various  sequences  of  activity. 
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From  the  EKG  data,  heart  rate  and  rate  variance  were  calculated  and  the 
amplitude  of  the  R  wave  of  the  EKG  complex  was  measured  and  its  variance  cal¬ 
culated.  The  EMG  data  were  analyzed  for  center  spectrum  frequency  and  total 
voltage.  The  tracking  data  were  analyzed  for  amplitude  and  variance. 

RESULTS 

In  the  fixed  base  simulator  experiments  the  pilots  flew  a  variety  of  man¬ 
ual  control  tasks,  tracking  air-to-air  and  air-to-ground  targets.  Various 
aspects  of  the  AFTI  software  were  introduced  one  by  one  in  semi -random  order 
and  their  effects  noted. 

Heart  rate  -  Figure  1  graphs  the  mean  heart  rates  of  the  subjects  for 
four  tracking  tasks  (air-to-air  and  ground)  as  speed  was  varied.  The  trends 
show  that  the  heart  rates  rose  as  the  speed  increased.  In  this  situation, 
the  physical  workload  remained  almost  constant  but  decisions  and  tracking 
effort  became  more  demanding.  Such  an  increase  in  mental  workload  was  shown 
by  the  small  rise  in  heart  rate. 

Heart  rate  variance  -  Figure  2  plots  the  heart  rate  variance  (root  mean 
square,  RMS)  of  the  subjects  grouped  with  speed  increase.  As  with  heart  rate, 
an  overall  rise  was  seen.  It  should  be  noted  that  the  downward  trend  of  task 
3  &  5  above  a  simulated  speed  of  350  kts  was  not  statistically  significant. 

R  wave  amplitude  -  The  amplitude  of  the  R  wave  for  the  subjects  grouped 
is  shown  in  Figure  3.  As  speed  increases,  the  R  wave  declines  overall.  Ad¬ 
mittedly,  tasks  6  &  7  showed  a  small  rise  over  a  narrow  speed  range.  The 
trend  for  tasks  8  &  9  failed  to  show  ahy  significant  difference  between  300 
and  500  kts  simulated  airspeed. 

R  wave  amplitude  variance  -  Figure  4  shows  how  R  wave  variance  altered 
with  speed  increase.  Overall,  an  increase  in  variance  was  observed. 

EMG  -  The  arm  electromyogram  recordings  showed  considerable  variability. 
With  three  of  the  tracking  tasks,  the  center  frequency  fell  as  speed  increased; 
with  the  fourth  no  change  was  observed.  The  data  recorded  from  the  neck  site 
were  corrupted  with  high  frequency  interference  from  switch  gear  in  the  simu¬ 
lator  and  could  not  be  used. 

Tracking  data  -  The  variance  of  the  tracking  data  also  showed  considera¬ 
ble  variability.  In  the  first  three  tracking  tasks  it  increased;  in  the  last 
it  showed  little  change. 

The  Dynamic  Environment  Simulator  (DES)  experiments  were  conducted  so 
that  the  pilots  performed  a  tracking  task  through  a  central  head  up  display 
while  the  direction  of  the  applied  acceleration  was  varied.  By  altering  the 
position  of  the  centrifuge  cab  on  the  arm  the  direction  of  acceleration  was 
varied  from  vertically  downwards  (+Gz),  to  lateral  to  the  left  (+Gy),  lateral 
to  the  right  (-Gy),  and  alternately  left  and  right  (+  Gy,  PMGy) . 

In  addition,  the  degree  of  physical  restraint  inside  the  cab  was  altered. 
The  shoulder  straps  were  fastened  and  the  inertia  reel  was  unlocked  (UNLOCK), 
the  inertia  reel  was  then  locked  (LOCK)  and  additional  shoulder  pads  (PADS) 
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were  placed  just  outside  of  the  shoulders  to  reduce  side  ways  movement  under 
lateral  acceleration  still  further.  The  first  DES  experiment  examined  these 
three  variants  under  the  different  acceleration  directions. 

Heart  rate  -  Figure  5  shows  how  the  heart  rate  varied  with  acceleration 
direction  and  restraint.  As  the  restraint  was  improved  (UNLOCK  -  LOCK  -  PADS) 
the  heart  rates  fell  considerably.  Note  that  the  extent  of  the  heart  rate 
change  is  very  much  greater  than  that  in  Figure  1,  where  speed  was  simulated 
in  a  fixed  base  simulator. 

Moreover,  as  the  acceleration  was  imposed  to  the  left  (+  Gy),  the  subjects 
were  forced  away  from  the  tracking  task  display  and  controls  and  greater  effort 
was  required  to  bring  the  head  and  body  back  to  the  center  of  the  cab.  This 
effort  produced  high  heart  rates  with  all  degrees  of  restraint.  Lateral  accel¬ 
eration  to  the  right  permitted  the  subjects  to  use  the  right  arm  rests  adjacent 
to  the  side  arm  controls,  thus  slightly  less  effort  was  needed  and  slightly 
lower  heart  rates  were  seen.  As  the  acceleration  alternated  from  right  and 
left,  the  varying  direction  required  less  sustained  effort  and  produced  lower 
heart  rates  still.  When  the  acceleration  was  vertical  (+  Gz),  no  effort  was 
required  to  acquire  the  display  and  controls  and  heart  rates  were  lowest. 

The  heart  rate  variance,  R  wave  and  variance  graphs  do  not  show  such 
clear  cut  changes  as  does  heart  rate.  For  these  three  variables,  neither  ac¬ 
celeration  direction  nor  degree  of  restraint  produced  significant  changes. 

The  second  DES  experiment  (II)  also  examined  variants  of  restraint  but 
the  unlock  mode  was  not  used.  Alternating  lateral  acceleration,  as  would  be 
experi  enced  during  tracking  in  the  AFTI  airplane,  was  used  rather  than  sus¬ 
tained  acceleration. 

Heart  rate  -  Figure  6  shows  the  data  from  two  subjects  both  with  and  with¬ 
out  PADS.  Data  are  presented  for  alternating  lateral  acceleration  (PMGy)  to¬ 
gether  with  pre  and  post  acceleration  controls  (static,  S  and  Baseline,  B). 

The  figure  shows  that  without  the  PADS,  the  heart  rates  were  significantly  ele¬ 
vated  by  lateral  acceleration.  With  the  PADS  this  pattern  was  not  seen;  the 
heart  rates  gradually  declined  over  the  course  of  the  experiment. 

Heart  rate  variance  -  Figure  7  also  shows  elevation  during  lateral  accel- 
eration  without  the  pads.  In  subject  M,  the  PADS  graph  shows  an  increase  of 
heart  rate  variance  with  acceleration  but  the  pattern  is  not  so  obvious  for 
the  NO  PADS  condition. 

R  wave  -  Figure  8  shows  no  deviations  with  acceleration  in  either  the 
PADS  or  NO  PADS  condition. 

R  wave  amplitude  variance  -  Figure  9  shows  significant  changes  with 
acceleration  in  the  NO  PADS  condition  while  the  PADS  curves  show  no  changes. 

The  graphs  for  EMG  (total  voltage  and  center  frequency)  show  no  changes 
with  acceleration.  The  tracking  task  data  (recorded  as  integrated  error  for 
a  single  pursuit  task  sequence)  do  show  that  tracking  was  worse  during  the 
acceleration,  particularly  so  when  shoulder  pads  were  absent  (Fig.  10).  Track¬ 
ing  during  the  control  periods  was  similar  for  both  conditions. 
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DISCUSSION 


Heart  rate  has  long  been  used  as  an  index  of  arousal,  effort,  and  work¬ 
load.  Many  factors  can  affect  heart  rate  but  if  these  are  controlled  and 
only  physical  or  mental  effort  is  altered,  their  effect  on  heart  rate  can  be 
examined.  In  the  fixed  base  simulator  studies,  the  subjects  were  well  rested, 
the  level  of  arousal  was  constant  and  little  physical  effort  was  required. 
Varying  the  speed  of  the  simulated  aircraft  and  targets  required  more  accurate 
tracking  and  faster  decisions.  This  increased  the  heart  rate  of  all  the  sub¬ 
jects.  Although  the  increases  were  small,  they  were  statistically  significant 
(p  <  .05  and  .001)  and  present  in  all  the  tasks  examined. 

Heart  rate  increases  were  also  seen  in  the  DES  experiments.,  Here  the 
mental  workload  was  constant,  the  task  did  not  change, neither  did  the  accel¬ 
eration  level,  only  the  direction  and  restraint.  When  sustained  lateral  ac¬ 
celeration  was  imposed,  the  subjects  were  forced  away  from  the  display  and 
controls  and  considerable  physical  effort  was  required  to  reacquire  them. 

This  produced  increases  in  heart  rate  but  of  a  level  considerably  higher  than 
those  in  the  fixed  base  simulator.  This  difference  is  a  measure  of  the  dif¬ 
ferential  response  to  physical  and  mental  workload.  The  worsening  of  re¬ 
straint  from  the  PADS  to  the  UNLOCK  mode  meant  that  the  subjects  had  to  work 
even  harder.  Similar  results  were  seen  in  the  second  DES  experiment.  Alter¬ 
nating  lateral  acceleration  required  slightly  less  physical  effort  when  the 
shoulder  pads  were  fitted  and  there  was  little  change  of  heart  rate  with  ac¬ 
celeration.  However,  without  the  pads,  acceleration  increased  the  heart  rate. 

The  use  of  heart  rate  variance  as  a  measure  of  workload  is  of  relatively 
recent  origin.  The  literature  on  the  changes  of  heart  rate  variance  with 
workload  is  conflicting.  In  these  studies,  heart  rate  variance  rose  as  speed 
increased  and  as  lateral  acceleration  and  poor  restraint  were  introduced. 
However,  the  changes  were  not  as  consistent  as  those  of  heart  rate  itself. 

The  R  wave  of  the  EKG  complex  is  associated  with  venticular  function. 

High  amplitudes  signify  increased  cardiac  output  and  greater  cardiac  effort. 
Normally,  as  heart  rates  increase,  the  R  wave  should  decline  in  amplitude 
as  individual  heart  beats  eject  smaller  volumes  of  blood.  The  more  frequent 
beats,  however,  contribute  to  greater  cardiac  output  .in  unit  time.  In  these 
studies,  R  wave  amplitude  declined  with  increase  in  speed  and  heart  rate  but 
showed  little  change  with  acceleration.  The  variance  of  the  R  wave,  however, 
was  most  useful,  it  rose  with  speed  and  acceleration,  significantly  so  when 
the  restraint  was  decreased. 

From  these  studies  it  can  be  seen  that  the  combination  of  increases  in 
heart  rate,  heart  rate  variance,  and  R  wave  variance  denote  an  increase  in 
workload  (physical,  mental  or  combined). 

The  EMG  data  were  disappointing.  Laboratory  studies  show  that  the  EMG 
can  indicate  muscular  effort  and  fatigue  by  examination  of  the  total  voltage 
and  center  frequency.  However,  laboratory  studies  involve  sustained  muscular 
effort  up  to  75%  of  maximum  and  for  long  periods  inducing  severe  fatigue.  In 
these  studies,  the  muscular  effort  was  slight  and  short  periods  only  were  re¬ 
quired,  so  it  is  not  so  suprising  that  no  significant  results  were  seen  in 
the  EMG  data.  Furthermore,  the  electrodes  and  leads  were  prone  to  interfer¬ 
ence  and  while  the  electrodes  were  non-invasive  (stick-on  skin  electrodes) 
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the  leads  on  the  arm  and  neck  did  prove  cumbersome.  Future  flight  studies 
will  not  involve  EMG  data  collection. 

Tracking  effort  has  been  used  repeatedly  in  the  past  to  measure  pilot 
performance.  Good  tracking  increases  the  times  for  and  chances  of  hits  on 
targets.  In  these  studies,  the  electrical  signals  recorded  as  tracking 
failed  to  show  significant  results.  However,  the  integrated  tracking  scores 
logged  by  hand  did  show  that  lateral  acceleration  decreased  the  scores,  par¬ 
ticularly  so  when  restraint  was  reduced.  It  may  be  that  the  signals  re¬ 
corded  from  the  simulator  computer  were  less  accurate  than  the  integration 
of  the  computer  itself. 

A  major  limitation  of  this  study  has  been  the  number  of  subjects. 
Clearly,  it  is  wise  to  restrict  access  to  such  a  valuable  airplane  as  the 
AFTI  F-16  but  this  has  meant  using  a  small,  highly- trained  group  of  subjects 
who  clearly  cannot  represent  the  pilot  population  as  a  whole.  However,  in 
the  event  it  has  proved  possible  to  derive  some  measures  of  physiology  and 
performance  that  can  indicate  workload  changes  and  these  will  be  used  in  fu¬ 
ture  flight  studies  to  compare  flight  regimes,  systems,  and  procedures  for 
workload  changes.  Furthermore,  the  individual  data  recorded  from  each  sub¬ 
ject  has  contributed  to  the  data  bank  that  can  be  used  when  those  subjects 
perform  in  flight  tests.  The  greater  value  of  the  data  recorded  may  well  ap¬ 
pear  in  later  studies. 


CONCLUSIONS 

Elevations  of  heart  rate,  variance,  and  R  wave  variance,  especially  when 
simultaneous,  indicate  conditions  of  increased  workload  for  the  subjects  used 
in  this  study.  This  should  be  true  for  inflight  studies  also. 

In  lateral  acceleration,  shoulder  pads  are  of  considerable  value.  They 
reduce  the  workload,  the  effort,  and  improve  tracking.  The  improvement  in 
heart  rate  alone  was  some  30  beats/min.  This  should  significantly  delay  the 
onset  of  pilot  fatigue  and  their  physical  limitations.  Shoulder  pads  should 
be  fitted  to  the  AFTI  F-16  and  all  other  aircraft  which  exhibit  sustained 
lateral  acceleration. 

EMG  data  are  not  so  reliable  in  simulated  flight.  This  combined  with 
the  additional  electrodes  and  leads  and  their  position  on  the  body  all  mili¬ 
tate  against  comfort  and  efficiency  and  will  be  eliminated  in  the  flight 
studies. 

Tracking  data  will  be  recorded  in  the  flight  phases  of  the  AFTI  F-16 
program.  In  the  hope  that  more  accurate  data  may  show  significant  effects  in 
the  future,  anal'yses  will  be  continued  to  check  whether  these  data  can  be 
used  to  measure  pilot  workload. 
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Figure  5  Mean  Heart  Rate  Average  and  Acceleration 
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Figure  8  Mean  R  Wave  Average  and  Accelerati 
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Figure  9  Mean  R  Wave  Variance  and  Acceleration  II 
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Monitoring  heart  rate  in  flight  is  a  relatively  simple  procedure;  the  technique 
is  not  intrusive,  it  does  not  compromise  flight  safety,  the  signals  are  easy 
to  record,  and  the  discrete  nature  of  the  data  make  them  amenable  to  various 
forms  of  analysis.  It  is  not  surprising,  therefore,  that  a  large  number  of 
experiments  have  been  reported  in  which  this  physiological  variable  has  been 
recorded  in  flight.  Although  most  of  these  experiments  were  designed  primarily 
to  examine  the  effects  of  various  physical  and  mental  stressors  on  pilots  a 
small  number  was  aimed  specifically  at  estimating  levels  of  workload  (1),  (2) 
and  (3) .  There  is  now  unequivocal  evidence  that  pilots  heart  rates  tend  to 
increase  during  flight  and  especially  during  such  demanding  manoeuvres  as  the 
take-off  and  the  landing. 

Using  heart  rate  to  estimate  workload  in  this  way  prompts  one  to  ask  a  number 
of  questions: 

1  What  is  the  relationship  between  a  pilot's  heart  rate  and  his  workload? 

2  Is  heart  rate  a  valid  and  reliable  indicator  of  workload? 

3  If  it  is  -  how  should  it  be  used? 

4  What  are  the  likely  neuro-physiological  mechanisms  involved? 

These  questions  will  be  discussed  using  examples  of  heart  rate  selected  from  more 
than  3000  plots  recorded  during  flight  trials  at  RAE  Bedford.  But  first  it  is 
important  to  describe  what  is  meant  by  the  term  pilot  workload.  There  are  many 
definitions  of  workload  most  of  which  appear  to  fall  into  2  broad  conceptual 
areas,  those  that  relate  to  the  task  or  to  the  demands  of  the  task  and  those 
that  are  associated  with  the  response  or  effort.  In  this  paper  workload  is 
considered  to  be  related  to  effort,  an  interpretation  which  is  compatible  with 
the  use  of  physiological  variables  and  which  also  lends  itself  readily  to 
subjective  assessment.  In  this  context  it  is  worth  noting  that  some  80%  of 
pilots  view  workload  as  being  effort-related  (4) ,  a  view  which  agrees  well  with 
the  influence  on  the  piloting  task  of  such  individual  factors  as  natural  ability, 
response  to  stress,  physical  fitness,  age,  training  and  experience. 

1  THE  RELATIONSHIP  BETWEEN  HEART  RATE  AND  WORKLOAD 

The  most  used  and  probably  the  most  reliable  methods  of  estimating  workload  in 
flight  are  those  based  on  some  form  of  subjective  reporting  by  experienced  test 
pilots.  And  so  it  is  of  interest  to  examine  the  relationship  between  pilots 
assessments  of  workload  and  their  heart  rate  responses. 

Following  a  3  year  exploratory  study,  in  which  heart  rates  were  recorded  from 
pilots  flying  a  wide  variety  of  aircraft,  it  was  decided  in  1972  to  monitor 
heart  rate  routinely  during  a  series  of  flight  trials  to  evaluate  different 
types  of  reduced  noise  landing  approaches  (5)  and  (6). 
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The  first  flight  trial  used  a  twin  turbo-prop  HS  Andover  to  compare  a  number  of 
different  approach  profiles  using  a  conventional  3°  glide  slope  as  a  datum. 
Single-segment  approaches  with  gradients  of  6°,  7^°  and  9°  and  two-segment 
approaches  with  a  7j°  slope  changing  to  3°  at  200  ft  were  studied  in  detail. 

Figure  1  shows  the  senior  project  pilot's  30  sec  heart  rates  for  the  single¬ 
segment  profiles  recorded  during  one  of  a  group  of  4  sorties.  The  experimenter 
used  a  Latin  square  design  to  allow  a  realistic  comparison  to  be  made.  Overall 
mean  heart  rates  for  the  4  approach  profiles  are  shown  in  Figure  2  for  the  same 
pilot.  In  this  case  there  was  exceptionally  good  agreement  between  heart  rate 
levels  and  subjective  estimates  of  workload;  and  also  with  expected  levels  of 
task  difficulty  -  the  workload  being  expected  to  increase  with  steeper  approach 
paths  and  higher  rates  of  descent. 

Later,  two-segment  approach  profiles,  with  a  slope  changing  to  a  3°  slope  at 
200  ft,  were  evaluated.  Figure  3  compares  mean  heart  rate  values  for  these 
2-segment  approaches  and  landings  with  those  for  3°  approaches  and  landings. 
Interestingly,  despite  their  relatively  low  heart  rate  responses  the  project 
test  pilots  initially  rated  the  workload  for  the  two-segment  approaches  as  high. 

It  later  transpired  that  these  2  pilots  had  instinctively  disliked  the  idea  of 
changing  from  a  steep  gradient  -  with  the  higher  rate  of  descent  -  to  a  normal 
gradient  at  200  ft.  After  the  first  sortie  they  modified  their  views  and  then 
consistently  rated  the  7|°/3°  approach  as  being  as  easy  as,  if  not  easier  than, 
the  normal  3°  approaches.  This  example  highlights  the  possibility  of  subjective 
assessments  of  workload  being  biased  by  allowing  instincts  and  misconceptions  to 
influence  judgement.  It  also  illustrates  the  advantage  of  using  heart  rate  to 
augment  -  or  sometimes  to  question  -  subjective  assessments  of  workload. 

In  a  later  trial  in  this  series  a  VC-10  four-jet  transport  was  used  to  evaluate 
5°/3°  two-segment  approaches  -  the  transition  from  the  steep  to  the  normal 
gradient  being  increased  to  500  ft  for  this  larger  aircraft.  Figure  4  illustrates 
beat-to-beat  heart  rates  recorded  from  the  handling  pilot  and  from  the  co-pilot 
during  an  early  two-segment  approach  and  landing.  The  introduction  of  beat-to-beat 
or  instantaneous  heart  rate  plots  increased  the  value  of  this  physiological  measure 
by  recording  short  tetm  changes  in  rate  which  can  be  used  to  identify  changes  in 
levels  of  workload.  For  example,  in  Figure  4  'A'  indicates  the  start  of  descent 
on  the  5°  glide  slope  -  in  this  case  at  a  greater  height  than  usual.  Points  'B' 
and  'C'  indicate,  respectively,  the  outer  marker  and  the  transition  at  500  ft. 

This  type  of  presentation  also  provides  a  bonus  measure  in  the  form  of  sinus 
arrhythmia. 

Figures  5  and  6  compare  overall  mean  30  sec  heart  rates  for  5°/3°  and  3°  approaches 
and  landings.  These  responses  confirmed  the  pilots  subjective  assessments  that 
the  two-segment  profiles  generated  similar  levels  of  workload  to  the  conventional 
3°  profile. 

These  examples  are  typical  of  flight  trials  in  which  different  experimental 
workload  levels  can  be  compared  in  a  realistic  way  with  a  convenient  datum  or 
with  each  other.  Throughout  the  series  there  was  a  substantial  measure  of 
agreement  between  relative  workload  levels  as  judged  by  pilots  subjective  estimates 
and  by  their  heart  rate  values.  Such  comments  made  in  later  discussion  as 
"...the  way  in  which  my  heart  rate  consistently  increased  at  that  point  reflects 
exactly  how  I  felt  about  the  difficulty  ..."  and  "I  was  aware  of  beginning  to  work 
harder  at  that  stage  of  the  approach  indicated  by  an  appreciable  increase  in  my 
heart  beat  ..."  are  typical. 
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A  number  of  other  flight  trials  at  Bedford  has  resulted  in  similar  levels  of 
agreement  between  subjective  estimates  of  workload  and  heart  rates.  For  example, 
in  trials  to  assess  the  value  of  a  "ski-jump"  take-off  technique  for  Harrier 
VSTOL  aircraft  heart  rate  responses  agreed  with  pilot  ratings  that  workload 
levels  for  these  take-offs  were  probably  less  and  certainly  no  greater,  than 
those  for  conventional  short  take-offs  in  this  aircraft  (7). 

During  a  recent  series  of  flights  to  evaluate  economic  category  3  landings  - 
consisting  of  autopilot  approaches  to  a  50  or  60  ft  decision  height  and  then 
a  manual  flare  and  touchdown  (8)  -  pilots  heart  rates  and  workload  ratings 
(using  a  10-point  scale)  for  the  decision  and  landing  were  recorded.  Figure  7 
is  a  typical  beat-to-beat  heart  rate  plot  showing  the  rapid  increase  in  workload 
as  decision  height  was  neared  and  manual  control  was  assumed  for  the  landing. 

The  scatter  diagram  (Figure  8)  illustrates  graphically  the  relationship  between 
32  heart  rate  responses  and  workload  ratings  in  real  fog  for  the  senior  project 
test  pilot.  These  data  varied  mojre  or  less  according  to  fog  conditions  and 
runway  visual  ranges  (RVRs) . 

Unlike  the  noise  abatement  trials  workload  levels  during  fog  approaches  arid 
landings  could  not  be  compared  directly  with  a  suitable  standard.  Nevertheless, 
heart  rates  recorded  in  fog  could  be  compared  indirectly  with  those  recorded 
during  approaches  and  landings  in  clear  weather.  Pilots  subjective  estimates  of 
workload  in  fog  tended  to  be  based  partly  on  comparison  with  those  in  clear 
weather  and  -  using  the  rating  scale  -  on  an  awareness  of  the  degree  of  spare 
capacity  available  for  other  tasks  (Figure  9) .  There  was  also  a  teridency  for 
pilots  to  compare  workload  levels  on  different  approaches  during  the  same  sortie 
as  fog  conditions  varied. 

Flight  trials  such  as  these  have  appeared  to  provide  strong  evidence  of  a 
reasonably  good  relationship  between  a  pilot's  heart  rate  response  and  his 
estimate  of  the  workload  level  associated  with  a  well  defined  and  demanding 
piloting  task.  And  it  is  a  relationship  that  appears  to  hold  good  both  for 
comparative  levels  of  workload  and  for  short  term  changes  in  workload  as 
indicated  by  changes  in  beat-to-beat  heart  rate. 

Unfortunately,  when  dealing  with  human  subjects  -  even  with  experienced  test 
pilots  -  discrepancies  and  inconsistencies  are  bound  to  occur  between  their 
opinions  and  their  heart  rate  responses.  In  most  such  instances  at  Bedford  a 
plausible  cause  for  the  disagreement  has  been  identified. 

2  HEART  RATE  AS  AN  INDICATOR  OF  WORKLOAD 

The  use  of  physiological  variables  to  indicate  levels  of  workload  has  been  viewed 
with  suspicion  by  many  people  and  the  use  of  only  one  variable  -  such  as  heart 
rate  -  has  been  criticised  in  particular.  However,  many  of  these  criticisms  have 
been  based  on  the  results  of  laboratory  and  flight  simulator  experiments  where 
often  the  task  and  levels  of  workload  were  unrealistic. 

Experience  at  Bedford  has  shown  that  when  the  pilot  is  in  the  handling  loop  - 
or  expecting  to  enter  the  loop,  and  when  the  flight  task  is  reasonably  demanding 
heart  rate  alone  will  usually  identify  meaningful  changes  and  differences  in 
workload.  Of  course,  expected  changes  in  workload  may  be  more  theoretical  than 
practical;  and  so  before  deciding  whether  heart  rate  can  differentiate  between 
workload  levels  it  is  important  to  be  sure  that  there  is,  in  fact,  a  real 
difference  (7). 
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When  the  flight  task  is  relatively  undemanding  or  when  the  pilot  is  in  a  purely 
monitoring  role  heart  rate  per  se  may  not  differentiate  between  small  differences 
in  workload.  But  often  in  these  instances  visual  inspection  of  beat-to-beat  plots 
will  reveal  changes  in  the  degree  of  heart  rate  variability  (sinus  arrhythmia) 
which  may  well  signify  changes  in  workload  (9). 

3  USING  HEART  RATE  TO  ASSESS  LEVELS  OF  PILOT  WORKLOAD  IN  FLIGHT 

When  monitoring  pilots  in  flight  it  is  obviously  desirable  to  obtain  their  active 
cooperation  and  it  is  even  better  to  have  their  enthusiastic  support  for  the 
technique.  At  Bedford,  test  pilots  frequently  apply  their  own  electrodes  before 
flight;  and  at  some  time  afterwards  it  is  quite  usual  for  them  to  express  a  keen 
interest  in  the  recorded  data  so  that  they  can  relate  their  subjective  impressions 
of  workload  and  task  difficulty  to  changes  in  their  heart  rate. 

Heart  rate  indicates  only  relative  differences  in  workload  and  so  it  is  helpful 
to  have  some  form  of  datum  for  purposes  of  comparison.  In  practice  assessment 
of  workload  is  usually  associated  with  the  introduction  of  a  new  aircraft  system 
or  operating  technique  and  so  one  can  often  compare  the  new  with  the  old. 

Although  it  is  not  always  possible  to  compare  heart  rate  responses  for  different 
experimental  variables  during  the  same  sortie,  or  even  under  similar  flight 
conditions,  the  advantages  of  doing  so  are  obvious. 

The  individual  nature  of  heart  rate  responses  make  it  almost  essential,  especially 
when  dealing  with  small  numbers  of  pilots,  for  each  pilot  to  be  considered  as 
his  own  control. 

A  pilot  may  compensate  for  an  easier  task  by  improving  his  performance  or, 
conversely,  he  may  allow  his  performance  to  deteriorate  rather  than  exert  more 
effort  to  meet  the  demands  of  a  more  difficult  task.  In  each  case  his  workload  - 
and  thus  his  heart  rate  -  may  remain  unchanged;  and  so  it  must  be  axiomatic  that 
when  assessing  workload  performance  criteria  are  clearly  defined  and  monitored. 

As  mentioned  earlier  differences  in  workload  are  more  likely  to  be  detected  by 
heart  rate  and  probably  by  subjective  assessment  when  the  task  is  realistically 
demanding.  And  so  the  technique  is  particularly  appropriate  for  estimating 
workload  during  the  approach  and  landing.  In  this  instance  the  task  is  well 
defined  and  performance  can  usually  be  monitored  by  on-board  instrumentation  and 
by  airfield-sited  kinetheodolites  or  radar. 

The  high  cost  of  operating  research  aircraft  usually  makes  it  impossible  to 
obtain  enough  data  for  worthwhile  statistical  analysis.  Nevertheless,  obvious 
trends  in  heart  rate  changes  can  be  used  in  conjunction  with  pilot  ratings  to 
provide  valuable  and  reliable  indications  of  differences  in  workload  levels. 
Surprisingly,  despite  being  more  used  to  obtaining  precise  measurements  from 
mechanical  and  electronic  devices,  trials  scientists  at  Bedford  have  found  pilots 
heart  rate  levels  and  subjective  ratings  to  be  of  definite  value  for  assessing 
or  comparing  levels  of  workload  in  flight. 

4  POSSIBLE  NEURO-PHYSIOLOGICAL  MECHANISMS 

There  is  a  substantial  amount  of  evidence  in  favour  of  workload  being  the  main 
determinant  of  heart  rate  levels  in  .experienced  pilots  during  demanding  flight 
manoeuvres.  It  is  interesting  to  speculate  on  possible  neuro-physiological 
mechanisms  that  would  explain  this  relationship.  Certainly,  it  is  rarely  due  to 
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physical  activity  -  which  during  normal  flight  is  very  low.  Although  the  fact 
that  heart  rates  are  higher  for  pilots  in  the  handling  loop  does  suggest  that 
increased  neuromuscular  activity  of  some  form  must  play  a  part. 

Piloting'  an  aeroplane,  especially  during  the  more  difficult  manoeuvres,  requires 
the  brain  to  collect,  filter  and  process  information  quickly;  to  exercise  judge¬ 
ment  and  make  decisions;  and  to  initiate  rapid  and  appropriate  actions.  This 
neurological  activity  -  which  must  have  been  essential  for  the  survival  of 
primitive  man  -  is  associated  with  a  state  of  preparedness  sometimes  known  as 
arousal.  Furthermore  there  is  experimental  evidence  that  increased  arousal  - 
up  to  a  moderate  level  -  enhances  a  person’s  capacity  for  complex  skills  and 
thus  improves  performance  (10).  For  instance,  Duffy  (11)  reported  that  the 
degree  of  arousal  "...  appears  to  affect  the  speed,  intensity  and  coordination 
of  responses  and  thus  to  affect  the  quality  of  performance".  She  also  observed 
that  in  general  the  optimum  level  of  arousal  appears  to  be  a  moderate  level  with 
the  curve  expressing  the  relationship  between  performance  and  arousal  taking  the 
form  of  an  inverted  'U'.  Other  authors  have  also  referred  to  such  a  relationship 
though  there  is  only  meagre  experimental  evidence  to  support  it  (12) .  Never¬ 
theless,  a  theoretical  relationship  of  this  type  has  a  particular  attraction  in 
the  context  of  flying  aeroplanes  as  there  is  evidence  that  both  under-  and 
over-arousal  have  preceded  landing  accidents  where  pilot  performance  was  clearly 
below  an  acceptable  level. 

There  is  some  experimental  evidence  that  a  similar  inverted  ' U'  shaped  function 
describes  the  relationship  between  performance  and  task  demands  (13) .  And  it 
has  been  suggested  that  levels  of  arousal  are  determined  by  task  characteristics 
or  demands,  by  how  the  individual  perceives  the  situation,  and  by  how  he 
responds  to  his  environment  (14)  (15).  And  so  one  can  speculate  that  a  pilot, 
by  matching  his  level  of  arousal  to  the  perceived  difficulty  of  a  flight  task, 
is  more  likely  to  produce  an  adequate  -  if  not  optimum  -  level  of  performance. 
The  result  will  depend  largely  on  his  training  and  experience,  although  if  the 
task  is  a  novel  one,  as  frequently  happens  in  test  flying,  a  significant  element 
of  empiricism  must  be  involved.  Clearly,  the  level  of  arousal  should  be  high 
enough  for  the  task  per  se  and  also  high  enough  to  allow  for  the  unexpected. 

For  example,  an  engine  failure  on  take-off  may  require  extremely  rapid  and 
appropriate  actions. 

On  occasions,  at  Bedford,  it  has  been  obvious  from  the  sudden  increase  in  heart 
rate  after  the  start  of  a  manoeuvre  that  a  pilot  had  failed  to  anticipate  the 
difficulty  of  the  task  and  "tune"  his  arousal  accordingly.  Conversely,  high 
heart  rates  have  been  recorded  -  both  before  and  during  a  manoeuvre  -  when 
there  was  an  element  of  uncertainty  about  the  task.  This  was  particularly 
noticeable  for  the  novel  "ski- jump"  take-offs  and  for  a  pilot's  first  approach 
and  manual  landing  in  fog.  The  probability  of  a  near  optimum  level  of  arousal 
being  generated  must  be  greater  if  a  pilot  has  recently  experienced  the  demands 
of  a  particular  flight  task.  Heart  rates  recorded  from  several  pilots  during 
sorties  of  approaches  and  landings  in  fog  became  lower  and  reasonably  consistent 
after  the  first  or  second  run.  Zwaga  (16)  in  describing  an  "adjustment  period" 
during  which  physiological  responses  to  specific  stimuli  were  moderated  related 
the  phenomenon  to  the  concept  of  arousal. 
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Anticipatory  increases  in  heart  rate  seen  before  the  start  of  manoeuvres  such 
as  the  take-off  presumably  indicate  an  increased  level  of  preparedness  in  the 
pilot's  neuro-physiological  system  for  the  demanding  task  to  follow.  It  seems 
clear  that  increasing  the  level  of  arousal  in  this  way  must  be  an  advantage  in 
the  same  way  that  sportsmen  "warm  up"  before  competitive  events.  In  other  words, 
a  task  requiring  a  high  level  of  psychomotor  skill  should  not  be  started  from 
cold. 

Support  for  these  speculations  is  provided  by  experimental  evidence  showing 
that  appropriate  pathways  in  the  brain  and  central  nervous  system  do  exist. 
Stimulation  of  the  reticular  activating  system  (RAS)  results  in  increased 
alertness,  improved  information  processing,  and  shorter  reaction  times.  This 
state  of  increased  arousal  is  apparently  sustained  by  reciprocal  feedback 
mechanisms  between  the  cortex,  the  RAS  and  the  hypothalamus.  The  hypothalamus, 
in  addition  to  regulating  autonomic  nervous  system  activity  -  which  includes 
heart  rate,  also  contains  integrating  and  organising  centres  concerned  with 
arousal. 

Although  the  concept  of  arousal  is  an  oversimplification  of  complex  neuro¬ 
physiological  mechanisms  it  is  functional  and,  providing  it  is  not  confused 
with  emotion,  it  conveniently  explains  the  relationship  between  a  pilot's 
workload  and  his  heart  rate. 

A  final  example  from  Bedford  may  confirm  -  in  a  practical  way  -  the  importance 
of  an  adequate  level  of  arousal  during  the  approach  and  landing.  Figure  10 
shows  two  beat-to-beat  heart  rate  plots  for  the  same  pilot  during  a  sortie  of 
6°  approaches  using  direct  lift  control  (DLC)  in  a  BAC  1-11.  The  upper  trace 
was  recorded  during  the  ninth  approach  which  ended  in  a  particularly  heavy 
landing  when  the  pilot  failed  to  arrest  the  rate  of  descent.  Damage  to  the 
aeroplane  necessitated  grounding  for  three  weeks.  Uncharacteristically  the 
heart  rate  did  not  increase  as  the  runway  threshold  was  neared  -  although  it 
increased  rapidly  after  the  hard  touchdown!  (The  temporary  interference  in 
the  trace  was  caused  by  the  jolt  on  landing  affecting  the  on-board  monitoring 
equipment).  The  lower  trace,  of  a  typical  response  for  a  6°  approach  and 
roller  landing,  is  shown  for  comparison. 

CONCLUSIONS 

The  questions  posed  at  the  beginning  of  this  paper  cannot  be  answered  with  any 
degree  of  scientific  certainty.  Nevertheless,  it  is  possible  from  the 
discussions  above  to  arrive  at  the  following  conclusions: 

1  There  is  good  evidence  that  heart  rate  responses  increase  with 
increased  workload. 

2  Differences  in  heart  rate  values  appear  to  indicate  relative 
differences  in  workload. 

3  Heart  rate  monitoring  is  best  used  in  conjunction  with  a  rating 
scale  for  workload. 

4  Although  the  exact  nature  of  the  neuro-physiological  mechanisms 
involved  is  not  known  it  is  possible  to  construct  a  reasonable  hypothesis 
using  the  concept  of  arousal. 

Reference  was  made  earlier  to  the  difficulty  of  defining  workload  -  perhaps  in 
the  interest  of  clear  thinking  one  should  avoid  using  this  term  altogether  and 
refer  instead  to  pilot  activity,  effort,  task  demands  and  so  on. 
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Fig  9  Pilot  workload  rating  scale 
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SOME  PHYSIOLOGICAL  CHANGES  IN  LATERAL  (Gy)  ACCELERATION 


Major  James  R.  Popplow,  M.D. 


Air  Force  Aerospace  Medical  Research  Laboratory- 
Aerospace  Medical  Division,  Air  Force  Systems  Command 
Wright-Patterson  Air  Force  Base,  Ohio 


ADDRESS  FOR  CORRESPONDENCE:  AFAMRL/BBS 

Wright-Patterson  AFB,  OH  45433 


ABSTRACT 


The  advent  of  the  Advance  Fighter  Technology  Integration  (AFTl) 
F— 1 6  has  prompted  a  renewed  interest  in  the  physical  and  physiological 
stresses  which  occur  in  lateral  acceleration.  The  Aerospace  Medical 
Research  Laboratories'  Acceleration  Effects  Branch  has  recently 
completed  this  experimental  protocol  which  examined  the  cardiopulmonary 
effects  of  Gy  forces. 


Eight  human  male  subjects,  who  were  all  permanent  members  of  the 
AFAMRL  centrifuge  subject  volunteer  group,  were  used  as  subjects.  They 
were  exposed  to  lateral  acceleration  forces  of  1-1/2,  2  and  2-1/2  Gy, 
both  left  and  right,  in  combination  with  either  1  or  2  Gz.  Recorded 
data  included;  peripheral  arterial  oxygen  saturation,  heart  rate, 
respiratory  rate  and  profile,  tracking  task  error  and  shoulder  pad 
pressures.  This  presentation  will  discuss  the  physiological  parameters. 


Our  preliminary  analysis  indicates  a  marked  reduction  in 
pulmonary  function  which  is  linearly  related  to  the  level  of  lateral 
force.  Using  only  these  modest  levels  of  Gy,  a  progressive  reduction  in 
peripheral  oxygen  saturation  and  a  concomitant  increase  in  heart  rate 
and  respiratory  rate  were  observed. 


These  results  are  contrasted  with  the  equivalent  levels  of 
vertical  (Gz)  and  transverse  (Gx)  accelerative  forces  and  a  discussion 
of  the  possible  adverse  effects  on  the  pilot  are  summarized. 
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PARAMETERS  MEASURED : 
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%  SATURATION 


Hemoglobin  Oxygen  Dissociation  Curve 
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SaC>2  and  Pulmonary  Shunt  Values  In  Acceleration 
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Figure  1.  Acceleration  Profile 
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FORCE  (POUNDS) 


200 


150 


100  S- 


50 


®  1  Gz 
A  2  Gz 


_ _L _ -JL 

1.5  2.0  2.5 


+  Gy 

LEFT  SHOULDER  PAD  FORCES  MEASURED  DURING  THE  LATERAL  G 
CENTRIFUGE  STUDIES.  MEANS  AND  STANDARD  DEVIATION  FOR  8  SUBJECTS. 


+oy 

+Gz 

POUNDS 

S.D. 

1.0 

0 

63 

+20 

1.5 

1 

83 

+19 

1.5 

2 

67 

+20 

2.0 

1 

123 

+28 

2.0 

2 

10lt 

+32 

2.5 

1 

lU6 

+U0 

2.5 

2 

13U 

+26 

TABLE  1.  Left  shoulder  pad  forces  measured  during  lateral  G  centrifuge 
studies.  Means  and  standard  deviation.  N=8. 
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INSP  TO  INSP 

INSP  VOL  (L)  TIME  (SEC)  EAR  OXIMETER  (%) 


TIME  (SEC) 


Figure  5 
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INSP  TIME  (SEC)  RESP  RATE  (RPM)  HEART  RATE  (BPM) 


If) 


(i/Yda)  3±va  iav3H 


o 

Ui 

a > 

LU 

1 

H 


10 


Figure  16 
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Figure  18 
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DUNCAN’S  MULTIPLE  RANGE  TEST*(ALPHA  =  0.05) 


Mean  Heart  Rate  at  Peak  GY/2GZ 
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MEAN  81.5 
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DUNCAN’S  MULTIPLE  RANGE  TEST  (ALPHA  =  0.05) 


Mean  Sa02  at  Peak  GY/2GZ 


TIME 


(SEC) 

GY 

MEAN 

2.5 

93.48 

2.0  “2.5 

93.95  94.05 

-2.0 
94.3  7 

-1.5 
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94.72 
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SaO^  Values  in  Lateral  Acceleration 


|  G 

|  Time 

i  Inspired 

1  Gas 

Minimum 

Sa02* 

S.D.  I 

..  _ Jj 

S  -1.5  GY  1  G 

30  SEC 

AIR 

93.9% 

1.0 

|  -1.5  GY  2  GZ 

30  SEC 

AIR 

93.5% 

1.4 

1  41.5  GY  1  GZ 

30  SEC 

AIR 

93.6% 

1.2  | 

I  41.5  GY  2  GZ 

30  SEC 

AIR 

93.0% 

2.0  1 

|  -2.0  GY  1  GZ 

30  SEC 

AIR 

93.5% 

1.5 

1  -2.0  GY  2  GZ 

30  SEC 

AIR 

93.0% 

1.1 

+2.0  GY  1  GZ 

30  SEC 

AIR 

92.5% 

1.5 

+2.0  GY  2  GZ 

30  SEC 

AIR 

91.9% 

1.2 

|  -2.5  GY  1  GZ 

30  SEC 

AIR 

92.9% 

1.7 

I  -2.5  GY  2  GZ 

30  SEC 

AIR 

92.2% 

1.2 

1  +2.5  GY  1  GZ 

30  SEC 

AIR 

92.0% 

1.2 

f  +2.5  GY  2  GZ 

30  SEC 

AIR 

91.7% 

1.2 

*  USING  95%  AS  ZERO  REF. 
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INTRODUCTION 

'  We  review  in  this  chapter  evidence  that 
suggests  that  the  Event-Related  Brain  Potential 
(ERP)  can  be  Incorporated  in  the  collection  of 
tools  of  Human  Engineering.  The  utility  of  the 
ERP  as  a  tool  in  the  study  of  Cognitive  Science 
has  been  discussed  elsewhere  (Donchin,  1975, 
1979,  1981;  Wickens,  1979).  As  the  Human  Factors 
that  must  be  addressed  by  the  engineer  are 
increasingly  “cognitive"  in  nature  (Rasmussen, 
1981;  Sheridan,  1981),  there  is  an  increasing 
need  for  enriching  the  repertoire  of  techniques 
for  the  assessment  of  cognitive  function.  We 
believe  that  psychophysiological  techniques,  in 
particular  ERP-based  procedures,  can  serve  this 
function.  We  realize  that  this  proposition  is 
not  self-evident  to  the  Human  Engineering 
profession.  The  recording  of  the  ERP  is 
cumbersome.  Electrodes  must  be  placed  on  the 
subject's  scalp.  Special  equipment  is  needed  for 
analyzing,  digitizing,  averaging  and  displaying 
the  data.  The  physiological  nature  of  the 
signals  Is  essentially  unknown  and  the  functional 
significance  of  the  ERP  components  is  a  subject 
of  controversy.  What  benefits  would  accrue  to 
the  system  designer  as  she  encumbers  herself  with 
this  exotic  technique?  Is  it  likely  to  help  in 
the  assessment  of  workload?  After  all,  there  is 
a  strong  tendency  to  trust  the  subjective  reports 
of  operators  in  assessing  workl oad.  These 
reports  appear  to  be  preferred  even  to  the 
seemingly  simpler  techniques  proposed  by  the 
Experimental  Psychologist.  Sheridan  and  his 
co-workers  concluded  (Sheridan  and  Simpson,  1979; 
Sheridan,  1980)  that  it  Is  possible  to  obtain  a 
reliable  and  valid  measure  of  workload  by 
administering  a  rather  simple  questionnaire.  Why 
should  one  bother  with  more  costly,  elaborate  and 
indirect  measurements  of  workload? 


Base,  California 
21,  1982 


The  question  is  reasonable  and  the  answer  is 
clear.  If  you  gain  nothing  by  complicating  the 
measurement  process  it  is  best  to  avoid  the 
complications.  We  claim,  however,  that  there  are 
circumstances  in  which  subjective  reports  may 
need  augmentation,  and  in  a  subset  of  these 
circumstances,  the  ERPs  may  be  very  useful. 

Consider,  for  example,  the  following  task. 
In  Figure  1  are  displayed  four  pairs  of  words. 
Your  task  is  to  write  "yes"  next  to  the  pair  if 
the  words  rhyme,  and  to  write  "no"  next  to  the 
pair  if  the  words  do  not  rhyme.  Please  do  so  as 
rapidly  as  you  can.  Did  you  find  it  equally  easy 
to  decide  in  all  four  cases?  Most  subjects 
report  that  the  decision  requires  the  same  effort 
regardless  of  the  pair  we  used  and  are  quite 
surprised  when  they  find  that  their  subjective 
assessment  of  the  workload  imposed  by  these 
simple  judgments  does  not  reflect  objective 
measures  of  performance. 

You  will  note  that  the  four  word  pairs  in 
Figure  1  are  instances  of  four  possible 
relationships  between  the  two  words  in  the  pair, 
as  follows: 

1.  (R0)  The  two  words  rhymed  and  looked 
alike  (MATCH-CATCH). 

2.  (R-)  The  two  words  rhymed  but  did  not 
look  alike  (MAKE-ACHE). 

3.  (WO)  The  two  words  looked  alike  but  did 
not  rhyme  (CATCH-WATCH). 

4.  (W-)  The  two  words  neither  rhymed,  nor 
looked  alike  (SHIRT-WITCH). 

We  label  these  pairs  with  an  R  to  indicate  a 
phonological  match,  with  an  0  to  indicate  an 
orthographic  match,  and  W  indicates  a 
phonological  mismatch  and  the  dash  (-)  an 
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Match  — Catch 
Make  —  Ache 
Catch  — Watch 
Shirt  —  Witch 

Figure  1.  A  sample  of  word  pairs  presented  to 
subjects  in  a  phonological  judgement  task. 


orthographic  mismatch.  Thus  for  the  RO  and  W- 
the  phonological  and  the  orthographic  information 
agree  and  for  the  R-  and  WO  pairs  there  is  a 
conflict  between  the  phonological  and  the 
orthographic  information.  While  it  is  easy  to 
analyze  the  stimuli  in  Figure  1  and  see  that  they 
do  indeed  differ  in  these  attributes,  subjects  do 
not  usually  perceive  themselves  as  having  greater 
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difficulty  in  deciding  that  the  words 
"CATCH-WATCH"  do  not  rhyme,  than  they  do  in 
deciding  that  the  pair  "SHIRT-WITCH"  do  not 
rhyme. 


But  these  subjective  impressions  are 
somewhat  misleading.  Polich,  McCarthy,  Wang  and 
Donchin  (in  preparation)  and  Kramer  and  Donchin 
(1982),  presented  subjects  with  the  two  words  of 
the  pair  in  succession  and  required  them  to 
indicate  their  judgments  by  pressing  one  of  two 
buttons  immediately  after  the  appearance  of  the 
second  word.  The  reaction  times  belie  the 
subjective  reports.  This  can  be  seen  from  Figure 
2  where  the  reaction  times  for  each  of  the 
classes  is  shown.  It  is  clear  that  a  conflict 
between  the  phonology  and  the  orthography  retards 
the  subject's  reaction  by  a  considerable  number 
of  milliseconds.  The  delay  is  about  300  msec 
when  the  second  word  "looks  like"  the  first  word 
but  does  not  rhyme  with  it  (the  WO  pairs  like 
CATCH-WATCH).  In  other  words,  an  individual's 
subjective  assessment  may  not  reveal  a  processing 
delay  that  may  cost  an  operator  up  to  three 
tenths  of  a  second  in  responding  to  a  display 
change!  Clearly,  a  non-trivial  delay  in 
high-performance  aircraft. 


What  we  find  then  is  that  when  tasks  place 
demands  on  the  human  information  processing 
system  that  affect,  or  depend  on,  interactions 


Ro  R-  Wo  W- 
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Figure  2.  Mean  Reaction  Times  for  correct  responses  and  percentage  of  errors  averaged  across  40  Subjects 
in. the  phonological  judgement  task  (After  Kramer,  et  al . ,  1982).  ' 
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between  the  automatically  activated  elements  of 
the  processing  machinery,  loads  may  be  imposed  on 
the  system  that  directly  affect  its  performance 
even  though  they  are  not  available  to  the 
internal  monitors  that  yield  subjective  reports. 

This  phase  of  the  analysis  illustrates  the 
need  to  supplement  subjective  reports  by  accurate 
and  detailed  measures  of  performance.  Where, 
though,  can  the  Psychophysiology  help?  We  submit 
that  its  most  effective  role  is,  when  properly 
used,  in  carrying  the  analysis  beyond  the  limits 
imposed  by  the  examination  of  the  more  overt 
responses.  Thus,  for  example,  the  data  in  Figure 
2  indicate  that  the  phonology-orthography 
conflict  delays  the  reaction.  But  these  data  do 
not  permit  unequivocal  conclusions  regarding  the 
functional  locus  of  the  delay.  Does  the  conflict 
cause  reprocessing  of  the  signal?  Are  the 
subjects  more  cautious  when  they  detect  the 
conflict,  or  do  they  require  more  time  to  encode 
the  stimuli?  Why  is  the  cost  of  conflict  lower 
for  the  R-  pair  than  it  is  for  the  WO  pair? 
These,  and  similar  questions,  are  important  not 
merely  for  their  theoretical  significance  but 
also  because  our  understanding  of  the  nature  of 
the  interference  is  necessary  if  we  are  to 
develop  systematic  guidelines  for  improving  the 
design  of  displays  and  related  systems.  The 
analysis  supported  by  the  ERPs  may  be  especially 
helpful  when  there  is  a  need  to  resolve 
conflicting  theories.  There  are  those  who 
suggest  that  phonologic  and  orthographic  codes 
interact  at  the  encoding  stage  (Meyer,  et  al . , 
1975;  Schulman,  et  al . ,  1978).  Others  have 
suggested,  that  the  interference  occurs  at  a 
response-selection  stage  (Conrad,  1978).  Kramer 
and  Donchin's  ERP  data,  shown  in  part  in  Figure 
3*  provide  information  that  compliments  the 
reaction  time  data. 

The  waveforms  shown  in  Figure  3  are  of  ERPs 
averaged  over  40  subjects.  These  data  were 
recorded  at  the  parietal  electrode,  and  each  of 
the  lines  represents  an  average  over  one  of  the 
four  classes  of  pairs  (R0,  R-,  WO,  W-).  As 
usual,  the  ERP  appears  as  a  sequence  of  peaks  and 
troughs,  (often  referred  to  as  "components").  It 
is  evident  that  the  waveforms  for  the  four  ERPs 
are  congruent  until  the  presentation  of  the 
second  stimulus.  The  subject,  of  course,  does 
not  know  which  of  the  four  pair-classes  will  be 
used  on  any  trial  until  the  appearance  of  the 
second  stimulus.  Once  this  happens  the  waveforms 
diverge.  It  is  quite  evident  that  the  ERP 
elicited  by  the  WO  pair  is  different  than  the 
other  three  ERPs.  It  is  characterized  by  a 
substantial  delay  in  the  elicitation  of  a  large 
positive  (down-going)  component  relative  to  the 
appearance  of  a  similar  component  in  the  other 
three  ERPs.  In  our  terminology,  the  latencyof 
this  peak,  labeled  the  "P300"  for  reasons  that 
will  become  apparent  later,  is  increased  in  the 
WO  pairs.  Thus,  the  ERP  provides  additional  data 
on  the  two  types  of  orthography-phonology 
conflict  that  occur  in  this  experiment.  If  we 
could  interpret  these  ERP  changes  we  may  be  able 
to  gain  a  better  understanding  of  the  process. 
In  fact,  since  the  latency  of  the  P300  component 
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Figure  3.  Average  ERPs  recorded  at  Pz.  The  ERPs 
were  averaged  across  40  subjects.  Data  span  an 
epoch  that  begins  100  msec  prior  to  the 
presentation  of  the  first  of  a  pair  of  words  in  a 
phonological  judgement  task  and  ending  2,560  msec 
after  the  second  stimulus  (After  Kramer,  et  al., 
1982). 


of  the  ERP  provides  a  measure  of  mental 
processing  time  that  is  unaffected  by  response 
selection  and  execution  processes  (McCarthy  and 
Donchin,  1981)  the  data  of  Figure  3  suggest  at 
least  some  of  the  effect  of  orthography-phonology 
conflict  operates  prior  to  the  response  selection 
stage.  It  is  interesting  to  note  that  the 
differences  in  the  reaction  time  are  larger  than 
the  P3O0  difference.  This  suggests  that 
interference  which  begins  prior  to  the  response 
selection  stage  is  amplified  during  later 
processing  and  therefore  may  reflect  a  cascading 
process  (McClelland,  1979). 

Further  examination  of  Figure  3  reveals  that 
the  four  ERPs  differ  also  in  the  disposition  of  a 
negative  (upwards-going)  peak  that  just  precedes 
the  P300.  This  peak  is  labeled  "N200".  The 
differences  in  the  amplitude  of  the  N200 
component  may  serve  to  clarify  some  issues 
concerning  the  detection  of  orthographic  and 
phonological  mismatches.  As  can  be  seen  in 
Figure  3,  the  largest  N200s  are  elicited  by  the 
W-  pairs,  in  which  both  the  orthography  and  the 
phonology  of  the  pair  members  mismatch.  The  R- 
1 i st  (phonological  match,  orthographic  mismatch) 
also  elicits  a  relatively  large  N200.  Thus  the 
R-  and  W-  list  which  both  orthographical ly 
mismatch,  elicit  an  N200.  This  suggests  that  the 
detection  of  an  orthographic  mismatch  may  occur 
automatically.  In  fact,  in  an  experimental 
condition,  not  shown  here,  the  subjects  were 
instructed  to  report  "yes"  if  the  words  match 
visually  regardless  of  the  phonology.  The  N200 
elicited  in  that  condition  by  R-  and  the  W-  was 
identical  to  that  elicited  during  the  rhyme 
condition.  On  the  other  hand,  the  WO  list 
(orthographic  match,  phonological  mismatch) 
elicits  an  N200  only  when  the  subject  is 
instructed  to  detect  rhymes.  This  suggests  that 
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a  phonological  mismatch  may  be  detected  only  when 
the  phonology  of  the  task  is  relevant.  In  other 
words,  the  ERP  data  indicate  that  a  phonological 
comparator  is  involved  solely  in  the  rhyme 
condition,  even  though  orthographic  comparators 
are  Involved  regardless  of  the  task.  Thus  both 
the  latency  of  the  P300  component  and  the 
amplitude  of  the  N200  component  provide 
Information  which  compliments  introspection  and 
traditional  overt  response  analysis.  The  ERP 
data  should  not  be  viewed  as  supplanting  the 
information  garnered  from  traditional  measures 
but  rather  as  a  source  of  complimentary 
information. 

We  discussed  these  data  because  they 
illustrate  our  basic  contention.  Subjective 
reports,  while  valuable,  do  have  limitations.  In 
assessing  the  demands  that  a  system  places  on  an 
operator  it  is  particularly  unwise  to  trust 
introspective  claims  that  deny  differences  in 
workload  between  the  systems  under  comparison. 
This  is  especially  so  when  the  demands  imposed  by 
the  system  operate  at  levels  of  processing  that 
are  not  normally  open  to  examination  by 
introspection.  It  is  in  this  domain  that  the 
Human  Factors  expert  is  most  likely  to  benefit 
from  the  models  and  techniques  of  the 
Experimental  Psychologist.  On  occasion  it  will 
be  found  that  the  assessment  can  be  augmented  by 
utilizing  ERPs.  This  is  particularly  true  when 
there  is  an  interest  in  developing  a  theoretical 
account  for  the  differences  between  the  demands 
Imposed  on  the  operator  by  different  systems. 
The  theoretical  models  that  can  be  adduced  abound 
in  references  to  internal  processing  entities. 
As  the  ERP  components  are  manifestations  of  such 
processing  entities  their  study  is  of  use. 

In  the  remainder  of  this  chapter  we  shall 
illustrate  these  concepts  by  reviewing  a  series 
of  studies  demonstrating  that  the  amplitude  of 
the  P300  can  serve  as  a  measure  of  "workload". 
We  shall  precede  this  discussion  with  a  brief 
Overview  of  the  study  of  ERPs.  For  more  details 
the  reader  is  referred  to  Callaway,  Tueting  and 
Koslow  (1978),  Otto  (1979),  and  Donchin  (in 
press). 

Introductory  Comments  on  the  P300  Component 

The  ERP  is  a  transient  series  of  voltage 
oscillations  in  the  brain  that  can  be  recorded 
from  the  scalp  in  response  to  the  occurrence  of  a 
discrete  event  (Donchin,  1975;  Regan,  1972).  The 
ERP  is  viewed  as  a  sequence  of  components 
commonly  labeled  with  an  "N"  or  a  "P"  denoting 
polarity,  and  a  number  which  indicates  their 
minimal  latency  measured  from  the  onset  of  the 
eliciting  event  (e.g.,  N100  is  a  negative  going 
component  which  occurs  at  least  100  msec  after  a 
stimulus).  Since  ERPs  are  relatively  small, 
relative  to  the  ongoing  EEG  (2  -  20mv  for  the 
ERPs  vs.  >50mv  for  the  EEG),  the  study  of  ERPs 
became  practical  only  after  the  development  of 
reliable  signal  averagers  (Clynes  and  Kohn, 
1959).  These  capitalize  on  the  fact  that  the  ERP 
is,  by  definition,  time-locked  to  the  eliciting 
event. 


It  is  crucial  to  recognize  the  componential 
nature  of  the  ERP.  Early  studies  of  the  ERP 
which  treated  the  waveform  as  a  unitary  entity, 
measuring  the  amplitude  over  the  entire  recording 
epoch  (Satterfield,  1965),  were  difficult  to 
interpret.  The  effects  of  the  experimental 
manipulations  tend  to  be  quite  specific  to  a  few 
components  and  a  combination  of  the  measures  of 
the  entire  epoch  may  obscure  the  relevant 
variance.  There  is  a  degree  of  controversy  as  to 
the  proper  identification  and  definition  of 
components,  (Donchin,  Ritter  and  McCallum,  1978; 
Picton  and  Stuss,  1980).  In  this  chapter, 
however,  we  shall  follow  Donchin  et  al's  (1978) 
definition  of  an  ERP  component  in  terms  of  the 
responsiveness  of  the  waveforms  to  specific 
experimental  manipulations.  A  component  is  thus 
mapped  into  a  cognitive  space  populated  by 
psychological  concepts  such  as  decisions, 
expectations,  plans,  strategies,  associations  and 
memories.  Specific  components  are  associated 
with  particular  entities  in  this  cognitive  space 
in  much  the  same  manner  in  which  cells  in  the 
periphery  of  the  visual  system  are  mapped  into  a 
field  in  the  visual  cortex.  The  subset  of 
elements  in  cognitive  space  associated  with  a 
particular  component  thus  contributes  to  the 
definition  of  the  ERP  component. 


The  specific  attributes  of  a  waveform  that 
are  examined  in  defining  a  "component"  are  the 
amplitude,  latency,  and  scalp  distribution.  It 
is  the  sensitivity  of  these  attributes  to 
experimental  manipulations  that  define  an  ERP 
component.  Although  no  reference  has  been  made 
to  the  underlying  neural  source  of  components,  it 
is  generally  assumed  that  a  scalp  distribution 
which  is  invariant  across  repeated  stimulus 
presentations  implies  a  specific  and  fixed  set  of 
neural  generators  (Goff,  Allison  and  Vaughan, 
1978;  Woods  and  Allison,  1981).  Thus  the  scalp 
distribution  which  is  related  to  the  underlying 
neural  population  responsible  for  the  generation 
of  the  component  is  assumed  to  be  a  crucial 
defining  characteristic. 


The  ERP  components  we  discuss  in  this  report 
are  "endogenous"  and  are  distinct  from  another 
class  of  ERPs  called  "exogenous"  (Sutton,  et  al . , 
1965;  Donchin,  et  al . ,  1978).  The  Exogenous 
components  represent  an  obligatory  response  of 
the  brain  to  the  presentation  of  a  stimulus. 
These  components  are  primarily  sensitive  to  such 
physical  attributes  of  the  stimuli  as  intensity, 
modality  and  rate.  The  seven  peaks  or  "bumps" 
which  occur  in  the  first  8-10  msec  after  the 
presentation  of  an  auditory  or  somatosensory 
stimulus  are  a  prototypical  example  of  the 
exogenous  category  (Jewett,  Romano  and  Williston, 
1970). 


The  endogenous  components,  on  the  other 
hand,  are  sensitive  to  the  processing  demands  of 
a  task  rather  than  the  physical  characteristics 
of  the  stimuli.  Endogenous  components, 
typically,  are  not  sensitive  to  changes  in  the 
physical  characteristics  of  the  eliciting 
stimuli.  On  the  other  hand,  these  components  are 
very  sensitive  to  changes  in  the  processing 
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demands  of  the  task  Imposed  on  the  subject.  The 
endogenous  components  are  nonobl igatory  responses 
to  stimuli.  The  strategies  and  expectancies  of 
the  subject  as  well  as  other  psychological 
aspects  of  the  task  account  for  the  variance  in 
the  endogenous  components.  A  typical  example, 
and  one  to  which  we  shall  devote  the  remainder  of 
this  chapter,  is  the  P300  component. 

This  ERP  component  is  elicited  by  rare,  task 
relevant,  stimuli.  A  task  in  which  it  is  readily 
elicited  is  often  called  the  "oddball"  paradigm. 
In  a  study  by  Duncan-Johnson  and  Donchin  (1977), 
using  this  paradigm,  the  subject  was  instructed 
to  count  covertly  the  total  number  of  higher 
pitched  tones  in  a  Bernoulli  series.  In 
different  blocks  of  trials  the  relative 
probability  of  the  two  tones  was  manipulated.  It 
can  be  seen  from  Figure  4  that  the  amplitude  of 
the  P300  increases  monotonically  as  the 
probability  of  the  stimulus  decreases.  This 
occurs  regardless  of  which  of  the  two  stimuli  is 
being  counted.  The  basic  relationship  between 
P300  amplitude  and  stimulus  probability  is  very 
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Figure  4.  Averaged  ERPs  elicited  by  high  and  low 
tones  presented  in  a  Bernoulli  series.  The 
waveforms  represent  experimental  conditions  in 
which  subjects  counted  the  high  tones  (solid 
lines)  or  solved  a  word  puzzle  as  the  tones  were 
presented  (dotted  lines)  (From  Duncan-Johnson  and 
Donchin,  1977). 


robust  and  has  been  replicated  numerous  times  in 
several  variants  of  the  classic  oddball  paradigm. 
The  only  requirement  is  that  the  subject  count 
one  of  the  two  events. 

When  the  subjects  were  solving  a  word  puzzle 
and  were  not  required  to  process  the  tones  the 
P300s  were  not  elicited.  Note  that  the  ERPs  in 
Figure  4  that  were  obtained  in  this  "ignore" 
condition  show  no  P300  at  all  levels  of 
probability.  Thus  the  amplitude  of  P300  is 
determined  by  a  combination  of  the  task  relevance 
and  the  subjective  probability  of  the  eliciting 
event.  This  basic  finding  plays  a  crucial  role 
in  the  use  of  P300  in  the  assessment  of  workload. 

The  demonstration  that  P300  is  elicited  by 
unexpected,  task  relevant  stimuli  led  Donchin, 
McCarthy,  Kutas  and  Ritter  (in  press)  to  suggest 
that  "the  P300  is  a  manifestation,  at  the  scalp, 
of  neural  action  that  is  Invoked  whenever  the 
need  arises  to  update  the  'neuronal  model' 
(Sokolov,  1969)  that  seems  to  underlie  the 
ability  of  the  nervous  system  to  control 
behavior".  The  neural  or  mental  model  is 
continually  assessed  for  deviations  from  inputs 
and  revised  when  the  discrepancies  exceed  some 
criterion  value.  The  frequency  with  which  the 
mental  model  is  revised  is  based  on  the  surprise 
value  and  task  relevance  of  the  stimuli.  Donchin 
(1981)  also  argued  that  the  concept  of  a 
subroutine  is  an  appropriate  metaphor  for  the 
activity  of  ERP  components  (Donchin,  Kubovy, 
Kutas,  Johnson  and  Herning,  1973;  Donchin,  1975). 
In  software  applications,  subroutines  represent 
algorithms  which  are  designed  to  accomplish  a 
specific  task  and  which  can  be  employed  in  a 
variety  of  different  programs.  ERP  components 
may  be  associated  with  specific  information 
processing  functions  which  are  activated  in  a 
variety  of  different  tasks.  In  the  case  of  the 
P300,  the  subroutine  may  be  invoked  whenever 
there  is  a  need  to  evaluate  surprising,  task 
relevant  events.  This  interpretation  of  the 
changes  in  P300  amplitude  is  strengthened  by  the 
evidence  that  has  accumulated  in  the  past  decade 
regarding  the  factors  that  control  the  latency  of 
the  P300.  As  the  use  we  make  of  P300  in  the 
analysis  of  man-machine  interaction  depends 
strongly  on  our  theoretical  interpretation  of  the 
component  it  will  be  useful  to  provide  a  brief 
review  of  the  latency  data  and  their 
interpretation. 


The  Latency  of  the  P300  Component 

The  peak  latency  of  the  P300  component 
appears  to  depend  on  the  time  required  to 
recognize  and  evaluate  a  task-relevant  event. 
The  latency  ranges  between  300  to  750  msec 
following  the  presentation  of  a  discrete 
stimulus.  In  fairly  simple  tasks  calling,  for 
example,  for  a  discrimination  between  two  tones 
that  differ  in  pitch  (i.e.,  1000-1600  Hz)  the 
stimuli  elicit  relatively  short  latency  P300s. 
More  difficult  discriminations  (i.e.,  semantic 
analysis)  result  in  increases  in  the  latency  of 
P300. 
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Assuming  that  manual  or  vocal  reaction  time 
terminates  processing,  and  that  P300  is  a 
manifestation  of  a  process  that  preceeds  the 
response  then  it  Would  be  expected  that  P300 
latency  and  reaction  time  should  positively 
covary.  This  prediction  has  been  supported  by 
numerous  studies  (Wilkinson  and  Morlock,  1967; 
Bostock  and  Jarvis,  1970,  Rohrbaugh,  Donchin  and 
Erikson,  1974).  Other  investigations,  however, 
failed  to  detect  a  relationship  between  P300 
latency  and  reaction  time  (Karlin,  Martz  and 
Morkoff,  1970;  Karlin  and  Martz,  1973). 

Donchin  et  al.  (1978)  proposed  an 
interpretation  of  the  processes  underlying  the 
P300  which  may  reconcile  these  contradictory 
findings.  They  suggested  that  P300  latency  is 
determined  by  the  time  required  to  evaluate  the 
stimulus,  but  is  largely  independent  of  response 
selection  and  execution  time.  The  correlation 
between  reaction  time  and  P300  latency  would, 
accordingly,  vary  as  a  function  of  the  percent  of 
reaction  time  variance  that  is  accounted  for  by 
stimulus  evaluation  processes.  This  percentage 
would  be  affected  by  the  strategies  employed  by 
the  subject.  The  strategies,  therefore,  should 
influence  the  relationship  between  P300  latency 
and  reaction  time  (see  also  Ritter,  et  al., 
1972).  Evidence  that  P300  is  determined  by  the 
amount  of  time  required  to  recognize  and  evaluate 
a  -  stimulus  has  been  reported  by  several 
investigators  who  employed  Sternberg's  (1966, 
1969a,  1969b)  additive  factors  methodology 
(Gomer,  Spicuzza  and  O'Donnell,  1976;  Ford,  Roth, 
Mohs,  Hopkins  and  Kopell,  1979;  Ford,  Mohs, 
Pfefferbaum  and  Kopell,  1980).  Sternberg's 
paradigm  involves  the  factorial  manipulation  of 
two  or  more  experimental  variables  which  are 
expected  to  differentially  affect  the  durations 
of  specific  stages  of  processing.  For  example, 
the  superimposition  of  a  mask  over  a  display  is 
assumed  to  influence  processing  in  an  early, 
perceptual  stage.  On  the  other  hand,  reduction 
of  the  compatibility  between  the  stimulus  and  the 
response  would  be  expected  to  affect  the 
selection  and  the  execution  of  the  response.  In 
the  studies  mentioned  above,  both  P300  latency 
and  reaction  time  increased  monotonically  with 
increasing  memory  load. 

Other  investigators,  employing  different 
paradigms  also  report  that  P300  latency  and 
reaction  time  are  positively  correlated  when 
stimulus  evaluation  time  is  manipulated.  N. 
Squires  et  al .  (1977)  found  that  P300  latency  and 
reaction  time  covaried  with  the  difficulty  of 
auditory  and  visual  discriminations. 
Furthermore,  P300  latency  varied  with  the 
manipulation  of  stimulus  discriminability  while 
reaction  time  was  influenced  by  both  stimulus 
evaluation  and  response  selection  factors. 
Heffley,  Wickens  and  Donchin  (1978)  performed  an 
experiment  in  which  subjects  were  required  to 
monitor  a  dynamic  visual  display  for 
intensifications  of  one  of  two  classes  of 
targets.  P300  latency  was  found  to  increase 
monotonically  with  the  number  of  elements  on  the 
display.  Since  subjects  were  not  required  to 
make  an  overt  response  the  differences  in  P-300 


latency  were  attributed  to  stimulus  evaluation 

processes. 

If  P300  latency  is  determined  by  stimulus 
evaluation  time  and  is  largely  independent  of  the 
time  required  for  response  selection  and 
execution,  then  experimental  variables  which  have 
a  different  effect  on  processing  time  in  the  two 
stages  should  influence  the  relationship  between 
P300  latency  and  reaction  time.  For  example, 
when  subjects  are  instructed  to  respond  quickly 
with  a  low  regard  for  accuracy,  their  responses 
are  probably  emitted  without  full  evaluation  of 
the  stimulus  (Wickelgren,  1977).  On  the  other 
hand,  if  subjects  are  instructed  to  respond 
accurately  they  are  likely  to  perform  a  more 
thorough  analysis  of  the  stimuli  prior  to 
responding.  This  analysis  leads  to  the 
prediction  that  the  correlation  between  P300 
latency  and  reaction  time  would  vary  with  the 
subject's  strategies.  Specifically,  the 
correlation  would  be  high  and  positive  when  the 
subjects  are  instructed  to  be  accurate.  Low 
correlations  would  be  observed  under  speed 
instructions. 

Kutas,  McCarthy  and  Donchin  (1977)  required 
subjects  to  distinguish  between  two  stimuli  under 
both  speed  and  accuracy  instructions.  In  one 
experimental  condition  subjects  were  required  to 
discriminate  between  two  names,  Nancy  and  David, 
presented  on  a  CRT  (with  relative  frequencies  of 
20  and  80%,  respectively).  In  a  second  condition 
female  names  comprised  20%  of  the  items  and  males 
names  80%.  In  the  third  condition  subjects  were 
required  to  discriminate  between  synonyms  of  the 
word  "Prod"  which  occurred  with  a  relative 
probability  of  20%  and  unrelated  words  which  were 
presented  with  the  complementary  probability. 
The  average  P300  latency  was  shortest  for  the 
first  condition,  intermediate  for  the  second  and 
longest  for  the  third  condition.  The  more 
complex  the  discrimination,  the  longer  the  P300 
latency.  A  detailed  analysis  of  the  single 
trials  (Woody,  1977)  revealed  that  the 
correlation  between  P300  latency  and  reaction 
time  was  larger  for  the  accuracy  condition  (.617) 
than  the  speed  condition  (.257).  Kutas  et  al . 
(1977)  concluded  that  the  data  supported  the 
hypothesis  that  P300  latency  reflected  the 
termination  of  a  stimulus  evaluation  process 
while  reaction  time  indexed  the  entire  sequence 
of  processing  from  encoding  to  response  selection 
and  execution.  Thus,  under  the  accuracy 
condition  when  response  selection  is  contingent 
on  stimulus  evaluation  processes  P300  latency  and 
reaction  time  are  tightly  coupled.  However,  when 
subjects  perform  the  discrimi nation  under  ’  the 
speed  instructions  the  processes  of  stimulus 
evaluation  and  response  selection  are  more 
loosely  coupled  and  hence  the  relationship 
between  P300  latency  and  reaction  time  is  not  as 
high. 

Additional  evidence  bearing  on  the  issue  of 
the  P300's  sensitivity  to  the  manipulation  of 
stimulus  evaluation  processes  has  been  obtained 
in  a  study  by  McCarthy  and  Donchin  (1980)  who 
manipulated  orthogonally  two  independent 
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variables  in  an  additive  factors  design 
(Sternberg,  1969).  One  factor,  stimulus 
discriminability,  has  been  shown  to  affect  an 
early  encoding  stage  of  processing  while  the 
second  factor,  stimulus  response  incompatibility, 
influences  the  later  stages  of  response  selection 
and  execution  (Bertlson,  1963;  Sanders,  1970; 
Shwartz,  Pomerantz  and  Egeth,  1977).  The 
subject's  task  was  to  decide  which  of  two  target 
stimuli,  the  words  RIGHT  or  LEFT,  were  presented 
in  a  matrix  of  characters  on  a  CRT.  The 
characters  were  either  presented  within  a  4x4 
matrix  of  #  signs  (no  noise  condition)  or  in  a 
4x4  matrix  of  letters  chosen  randomly  from  the 
alphabet  (noise  condition).  Stimulus  response 
incompatibility  was  manipulated  by  preceeding  the 
target  matrix  either  with  the  cue  SAME  or  with 
the  cue  OPPOSITE.  SAME  signaled  a  compatible 
response.  The  cue  OPPOSITE  indicated  an 
incompatible  response;  the  right  hand  would 
respond  to  the  word  LEFT  and  the  left  hand  to  the 
cue  RIGHT.  Reaction  time  increased  when  the 
command  word  was  embedded  in  noise  and  when  the 
response  was  incompatible  with  the  stimulus.  The 
effect  of  the  two  variables  on  the  RT  was 
additive  implying  that  these  manipulations 
influenced  different  stages  of  processing.  P300 
latency  was  increased  by  the  addition  of  the 
noise  to  the  target  matrix,  but  was  not  affected 
by  the  incompatibility  between  the  stimulus  and 
the-  response.  These  results  support  the 
conclusion  that  P300  latency  is  affected  by  a 
subset  of  the  set  of  processes  which  affect 
reaction  time.  The  P300  is  elicited  only  after 
the  stimulus  has  been  evaluated.  Subsequent 
processing  required  for  the  selection  and 
execution  of  the  response  does  not  appear  to 
influence  the  latency  of  the  P300. 

The  P300  component  of  the  ERP  provides  a 
metric  for  the  decomposition  of  stages  of 
information  processing  which  compliments  the 
traditional  behavioral  measures.  In  terms  of 
applications  to  system  design  and  workload 
evaluation  ERPs  used  in  conjunction  with 
behavioral  and  subjective  measures  permit  the 
assessment  of  stage  specific  task  interference 
effects.  For  example,  if  two  time-shared  tasks 
interfere  with  each  other  it  is  usually  desirable 
to  know  the  locus  of  this  interaction.  Only  by 
discovering  the  stage  at  which  tasks  interact  can 
systems  be  designed  which  minimize  operator 
workload. 


THE  P300  AND  HUMAN  ENGINEERING  ' 

P300  and  Perceptual /Central  Processing  Resou rces 

The  studies  reviewed  above  provided  evidence 
that  the  P300  component  is  a  manifestation,  at 
the  scalp,  of  a  processing-entity,  or  a 
subroutine,  that  is  involved  whenever 
task-relevant  surprising  stimuli  are  present. 
The  routine  appears  to  be  performing  a  role  in 
the  context-updating  activities  that  occur 
whenever  an  event  calls  for  the  revision  of  the 
neuronal  model  or  schema  of  the  environment.  It 
is  noteworthy  that  this  subroutine  is  invoked 


only  if  the  stimuli  are  associated  with  a  task 
that  requires  that  they  be  processed.  Ignored 
stimuli  do  not  elicit  a  P300.  But  what  if  the 
stimuli  are  only  partially  ignored?  What  if  the 
subject  is  instructed  to  perform  the  oddball  task 
concurrently  with  another  task?  Would  the 
amplitude  of  the  P300  reflect  the  centrality  of 
the  eddball  task?  Would  it,  perhaps,  change  with 
the  amount  of  resources  allocated  to  the  oddball 
task?  Clearly,  if  so  -  the  P300  may  serve  as  a 
very  useful  measure  of  the  amount  of  resources 
demanded  by  the  two  tasks.  It  is  this  series  of 
questions  that  lie  at  the  core  of  the  usage  that 
can  be  made  of  P300  in  the  assessment  of 
workload. 

The  study  of  cognitive  workload  and  of  the 
allocation  of  processing  resources  to  several 
tasks  performed  concurrently  is,  in  fact,  the 
area  of  research  that  has  profited  from  the 
incorporation  of  ERP  measures.  The  research 
reviewed  here  began  in  the  Cognitive 
Psychophysiology  Laboratory  in  the  mid  1970's 
with  support  from  DARPA.  It  has  been  performed 
within  the  framework  of  Resource  Allocation 
theory.  This  class  of  models  suggests  that  it  is 
useful  to  conceptualize  human  capacity  as 
represented  by  a  finite  pool  of  "resources" 
available  for  sharing  among  concurrently 
performed  tasks  (Kahneman,  1973;  Moray,  1967; 
Norman  and  Bobrow,  1975).  In  the  Kahneman  (1973) 
model  these  processing  resources  were 
undifferentiated,  implying  that  all  tasks  draw 
resources  from  the  same  pool.  The  general  model 
predicts  that  when  two  tasks  are  timeshared  their 
performance  should  decrease  relative  to  single 
task  levels. 

This  model  underlies  the  secondary  task 
technique,  a  method  which  is  commonly  employed  in 
the  assessment  of  the  workload  associated  with  a 
task  as  the  workload  is  viewed  as  reflected  by 
the  amount  of  processing  resources  consumed  by  a 
task  (Knowles,  1963;  Rolfe,  1971;  Wickens,  1979). 
In  the  secondary  task  technique,  the  subject  is 
assigned  two  tasks;  a  "primary"  task  that  is  to 
be  performed  as  well  as  possible  and  the  second 
task  that  need  be  performed  only  to  the  extent 
that  primary  task  performance  remains  stable.  It 
is  assumed  that  the  demands  imposed  upon  the 
subjects  by  the  primary  task  can  be  assessed  by 
monitoring  performance  on  the  secondary  task.  An 
easy  primary  task  will  require  a  minimal  amount 
of  processing  resources,  leaving  an  ample  supply 
for  the  performance  of  a  secondary  task,  while  a 
difficult  primary  task  will  require  the  majority 
of  processing  resources,  leaving  an  insufficient 
supply  for  the  performance  of  the  secondary  task. 
Thus,  the  better  the  performance  of  the 
secondary  task  the  less  demanding  the  primary 
task. 

Although  the  secondary  task  procedure  has 
been  extensively  used  it  presents  a  number  of 
practical  problems  (Brown,  1978;  Ogden,  Levine 
and  Eisner,  1979).  Particularly  unfortunate  is 
the  fact  that  secondary  task  responses  often 
intrude  upon  primary  task  performance.  Of 
course,  fluctuations  in  primary  task  performance 
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make  the  Interpretation  of  the  resource  trade-off 
extremely  difficult.  Evidently,  it  would  be 
useful  to  have  a  secondary  task  which  is 
sensitive  to  changes  in  primary  task  difficulty 
but  which  does  not  require  an  overt  response. 

It  was  the  basic  assumption  of  our  research 
program  that  the  oddball  task  can  be  used  as  a 
non-intrusi ve  secondary  task  since  the 
ERP-eliciti  ng  tones  occur  intermittently,  are 
easily  discrimi nable  and  do  not  require  an  overt 
response.  Another  advantage  of  this  procedure  is 
that  it  could  be  applied  uniformly  across 
different  operational  settings.  In  other  words, 
the  oddball  task  could  be  inserted  into  virtually 
any  operational  setting  without  requiring 
modifications  in  the  system  associated  with  the 
primary  task.  Wickens,  Isreal  and  Donchin  (1977) 
reported  one  of  the  first  studies  in  the  series 
using  a  compensatory  tracking  task  as  the  primary 
task  and  the  oddball  paradigm  as  the  secondary 
task. 

Figure  5  illustrates  the  experimental 
procedures  used  in  this  and  several  other  studies 
to  be  discussed.  The  subjects  sat  in  front  of  a 
CRT  and  were  instructed  to  cancel  computer 
generated  cursor  movements  by  keeping  the  cursor 
superimposed  on  a  target  in  the  center  of  the 
display.  This  was  accomplished  by  movement  of  a 
joystick  mounted  on  the  right-hand  side  of  the 
subject's  chair.  Levels  of  tracking  difficulty 
were  manipulated  by  requiring  the  subject  to 
track  in  either  one  or  two  dimensions  (horizontal 
and/or  vertical).  The  compensatory  tracking  task 
was  defined  as  the  primary  task.  In  addition  to 
the  tracking,  the  subjects  were  also  instructed 
to  count  one  -of  two  tones  presented  in  a 
Bernoulli  series  of  high  and  low  pitched  tones. 
Control  conditions  were  also  included  in  which 
the  subjects  performed  each  of  the  two  tasks 
separately. 

The  data  indicate  that  the  introduction  of 
the  tracking  task  drastically  diminishes  the 


amplitude  of  the  P300.  However,  no  further 
reduction  in  P300  amplitude  could  be  observed  as 
tracking  difficulty  increased  by  requiring 
tracking  in  two  dimensions.  Even  though  tracking 
difficulty,  assessed  by  Root  Mean  Square  error 
(RMS),  as  well  as  by  reaction  time  to  the  tones, 
definitely  increased  with  the  addition  of  a 
tracking  dimension,  P300  amplitude  did  not 
change.  Isreal,  Chesney,  Wickens  and  Donchin 
(1980)  conducted  a  similar  study  requiring 
subjects  to  perform  a  compensatory  tracking  task 
concurrently  with  a  counting  task.  In  this  case, 
however,  the  bandwidth  of  the  random  forcing 
function  rather  than  the  dimensionality  of  the 
tracking  task  was  manipulated.  The  bandwidth  was 
increased  gradually  until  the  cursor's  speed 
reached  the  highest  level  the  subject  could 
tolerate  without  exceeding  a  preset  error 
cri  teri  on. 

The  results  are  shown  in  Figure  6.  Again, 
P300  amplitude  is  diminished  by  the  introduction 
of  the  tracking  task,  but  increases  in  the 
bandwidth  of  the  forcing  function  did  not  produce 
systematic  changes  in  the  amplitude  of  the  P300. 
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Figure  6.  Average  parietal  ERPs,  elicited  by 
equiprobable  counted  tones,  for  each  bandwidth 
interval  and  count-only  control  conditions,  for 
both  ascending  and  descending  blocks  of  trials 
(From  Isreal,  et  al . ,  1980). 


Figure  5.  An  illustration  of  the  experimental’ 
paradigm  employed  in  the  analysis  of  the  utility 
of  the  ERP  as  a  workload  measure. 
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These  results  cannot  be  explained  easily  within 
the  framework  of  an  undifferentiated  capacity 
theory  if  we  assume  that  P300  amplitude  indexes 
the  demands  placed  on  the  subject  by  the  primary 
task.  Increasing  the  bandwidth  clearly  affects 
the  performance  of  overt  secondary  tasks 
(McDonald,  1973;  Wierwille,  Gutmann,  Hicks  and 
Muto,  1977).  The  fact  that  P300  did  not  change, 
even  though  a  dramatic  drop  in  amplitude  was 
observed  with  the  introduction  of  the  task, 
required  explanation. 

One  interpretation  of  the  results  is  that 
the  P300  is  not  sensitive  to  the  processing 
demands  of  the  task  but  instead  reflects  the 
motor  activity  required  by  tracking.  The 
hypothesis  was  tested  by  Isreal  et  al.  (1980)  who 
instructed  subjects  to  manipulate  a  joystick  with 
one  hand  concurrently  with  the  oddball  task.  The 
amplitude  of  the  P300  component  elicited  by  the 
tones  was  not  affected  by  the  motor  demand. 
Thus,  it  would  seem  that  hand  movements,  per  se, 
did  not  decrease  the  amplitude  of  the  P300. 

Another  interpretation  of  the  results  is 
that  the  resources  that  are  tapped  when  the 
dimensionality,  or  the  bandwidth  of  the  target, 
are  increased  are  not  the  resources  required  by 
the  oddball  task.  Several  investigators  have 
proposed  that  processing  resources  are  not 
undifferentiated  but  are  rather  structured 
according  to  various  information  processing 
stages  (Kantowitz  and  Knight,  1976;  Kinsbourne 
and  Hicks,  1978;  Navon  and  Gopher,  1979,  1980; 
Sanders,  1979).  Wickens  (1980)  has  identified 
hypothetical  processing  structures  on  the  basis 
of  input  and  output  modalities  (visual -auditory, 
manual -vocal ),  stages  of  information  processing 
(encoding  and.  central  processing,  response 
selection  and  execution)  and  codes  of  processing 
(verbal,  spatial).  In  this  framework  dual -tasks 
are  expected  to  interfere  to  the  extent  that  they 
share  overlapping  resources.  For  example,  two 
tasks  which  both  require  substantial  central 
processing  wil 1  interfere  with  each  other  to  a 
greater  extent  than  a  task  with  central 
processing  demands  and  another  with  heavy  demands 
for  response  processes.'  This  view  of  the 
allocation  of  processing  resources  is  consistent 
with  studies  which  show  little  or  no  decrement  in 
performance  when  two  difficult  tasks  are 
time-shared  (Allport,  Antonis  and  Reynolds,  1972; 
North,  1977;  Wickens  and  Kessel,  1979). 

The  notion  that  P300  is  sensitive  to  a 
specific  aspect  of  information  processing  is 
consistent  with  the  data,  reviewed  above, 
regarding  the  relation  between  P300  latency  and 
reaction  time.  P300  latency  appears  to  be 
sensitive  to  a  subset  of  the  processes  that 
determine  reaction  time.  Furthermore,  P300 
latency  is  influenced  by  manipulations  of  factors 
which  are  assumed  to  affect  relatively  early, 
stimulus  evaluation  processes  while  being 
insensitve  to  changes  in  variables  which  produce 
their  effect  on  the  later  response  selection  and 
execution  processes.  If  the  manipulation  of  the 
dimensionality  and  bandwidth  of  the  tracking  task 
demand  resources  associated  largely  with  response 


selection  and  execution  processes  then  P300 
amplitude  should  not  reflect  fluctuations  in 
performance.  On  the  other  hand,  if  the 
perceptual  aspects  of  a  task  were  manipulated, 
the  amplitude  of  the  P300  elicited  by  a  secondary 
task  would  be  expected  to  covary  with  primary 
task  difficulty. 

Isreal,  Wickens,  Chesney  and  Donchin  (1980) 
tested  the  latter  hypothesis  by  combining  the 
oddball  task  as  a  secondary  task  with  a  visual 
monitoring  task  that  served  as  the  primary  task. 
The  subjects  were  instructed  to  monitor  a 
simulated  air  traffic  control  display  either  for 
course  changes  or  for  intensifications  of  one  of 
two  classes  of  stimuli  (triangles  or  squares). 
Primary  task  difficulty  was  manipulated  by 
increasing  the  number  of  elements  traversing  the 
CRT  (Sperando,  1978).  The  numerosity  variable 
did  in  fact  have  a  systematic  effect  on  reaction 
time  to  the  tones  when  subjects  were  monitoring 
for  course  changes.  Reaction  time  increased 
monotonically  from  the  control  condition  to  the 
condition  in  which  subjects  were  required  to 
monitor  eight  elements  simultaneously.  However, 
in  the  flash  detection  condition  reaction  time 
did  not  increase  significantly  as  a  function  of 
the  number  of  elements  displayed. 

As  can  be  seen  from  Figure  7  the  P300 
elicited  by  the  counted  tones  decreased 
monotonically  with  increases  in  difficulty  in  the 
monitoring  task  when  subjects  were  detecting 
course  changes.  In  the  flash  detection  condition 
P300s  decreased  with  the  introduction  of  the 
monitoring  task,  but  increases  in  the  number  of 
display  elements  failed  to  further  attenuate  P300 
amplitude.  This  result  is  also  consistent  with 
the  reaction  time  data.  Since  the  primary  task 
did  not  require  a  response,  the  data  of  Isreal  et 
al.  (1980)  have  demonstrated  that  P300  amplitude 
is  sensitive  to  the  perceptual  demands  of  a 
primary  task. 

The  Use  of  P300  in  Task  Analysis 

This  structure-specific  conception  of 
processing  resources  has  several  implications  for 
the  study  of  man-machine  systems.  One  area  which 
might  benefit  from  the  use  of  the 
structure-specific  analysis  of  human  information 
processing  resources  is  task  analysis. 
Traditionally,  the  analysis  of  operator 
performance  in  complex  systems  has  been  conducted 
by  detailing  the  observable  aspects  of  tasks  and 
task  sequences  (Kidd  and  Van  Cott,  1972).  This 
analysis  has  usually  taken  the  form  of  elaborate 
flow  charts  which  outline  aspects  of  operator 
behavior  such  as  information  input,  decisions  and 
required  actions  (Coakley  and  Fucigna,  1955; 
Folley,  et  al.,  1960).  Although  these  procedures 
provide  an  accurate  description  of  the  behavior 
exhibited  by  the  operators,  they  do  not  enable 
micro  analysis  of  the  task  that  could  provide  the 
system  designer  with  information  on  the  resources 
required  by  different  sub-task  sequences.  It 
would  be  useful  to  examine  a  breakdown  in 
performance  under  high  workload  conditions  for 
their  relation  to  resource-competition.  For 
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Figure  8.  The  temporal  sequence  of  the  target 
acquisition  task  (from  upper  right  to  lower 
left).  The  large  element  is  the  computer 
controlled  target  while  the  smaller  element 
represents  the  subject  controlled  cursor.  The 
joy  stick  on  the  right  side  regulates  the  path  of 
the  cursor  in  the  X  and  Y  axes.  The  joy  stick  on 
the  left  controls  the  rotational  velocity  of  the 
cursor  (From  Wickens,  et  al . ,  1982). 


Fi pure  Single  subject  and  average  ERPs 
elicited  by  infrequent,  counted  tones  presented 
concurrently  with  each  of  two  monitoring  tasks. 
Two  monitoring  conditions  as  well  as  a  count-only 
control  condition  are  presented.  All  waveforms 
displayed  were  recorded  at  Pz  (From  Isreal ,  et 
al.,  1980). 


example,  it  would  be  useful  to  know  if  the 
operator  is  required  to  perform  tasks  which 
demand  a  great  deal  of  response  processing  but 
little  perceptual  analysis. 

Wickens,  Kramer  and  Donchin  (1982)  performed 
a  componential  analysis  of  the  demands  of 
controlling  higher  order  systems,  well  validated 
in  the  literature  to  impose  a  greater' load  on 
information  processing  resources  (Baty,  1971; 
Fuchs,  1962).  By  "order  of  control"  we  refer  to 
the  number  of  time  integrations  of  the  output  of 
a  controller  (i.e.,  joystick)  and  the  output  of 
the  system.  In  a  first  order,  or  velocity  driven 
system,  a  deflection  of  the  joystick  corresponds 

to  a  change  in  the  velocity  of  the  controlled 
element.  A  second  order,  or  acceleration  driven 
system,  produces  a  change  in  the  accleration  of 
the  controlled  element  proportional  to  the 
movement  of  the  control  stick.  Assuming  that 
P300  amplitude  is  sensitive  to  the  perceptual 
aspects  of  a  task  then  a  reduction  in  P300 
amplitude  by  higher  order  control  should  localize 


the  influence  of  the  order  variable  at  the 
earlier  processing  stages. 

Figure  8  illustrates  the  subject's  task. 
The  target  appeared  on  the  screen  and  moved  in  a 
straight  line,  but  at  a  randomly  selected  angle, 
in  the  direction  of  its  exit.  The  subject  had  to 
move  the  cursor  into  the  neighborhood  of  the 
target.  The  time  between  the  appearance  of  the 
target  and  its  acquisition  by  the  cursor  is 
called  the  "acquisition  phase".  Acquisition  was 
accomplished  by  manipulating  the  two-axis 
joystick  mounted  on  the  right  side  of  the 
subject's  chair.  Successful  acquisition 
initiated  the  alignment  phase.  The  target  began 
to  rotate  at  a  constant  velocity  in  either  a 
clockwise  or  counterclockwise  direction.  The 
subjects  had  to  rotate  the  cursor  at  the  same 
velocity  as  the  target  while  also  keeping  the  two 
elements  superimposed.  The  rotation  was 
accomplished  by  manipulating  the  single  axis 
joystick  mounted  on  the  left  side  of  the 
subject's  chair.  A  deflection  of  the  stick  to 
the  right  produced  a  clockwise  rotation  of  the 
cursor  at  an  angular  velocity  proportional  to  the 
angle  of  deflection,  a  deflection  to  the  left 
produced  a  counterclockwise  rotation.  Deviation 
from  the  initial  acquisition  criterion  for  more 
than  1000  msec  necessitated  a  re-alignment  of  the 
elements.  Once  the  subject  decided  that  all  of 
the  criteria  had  been  satisfied  and  the  target 
and  cursor  were  aligned,  she  could  press  a 
capture  button  and  the  trial  was  terminated. 


380 


We  assumed  that  the  alignment  phase  would  be 
more  difficult  than  the  acquisition  phase  due  to 
increased  perceptual  '  demands  imposed  by  the 
requirement  to  control  the  additional  rotational 
axis.  We  predicted,  therefore,  that  the  P300 
amplitude  elicited  by  the  tones,  associated  with 
an  oddball  task  run  concurrently  with  the 
tracking  task,  would  be  larger  during  the 
acquisition  than  during  the  alignment  phase. 

The  ERP  results  presented  in  Figure  9 
confirm  these  predictions.  The  P300  amplitude  is 
attenuated  both  as  a  function  of  phase,  larger 
amplitude  P300s  being  elicited  in  the  acquisition' 
phase,  and  system  order,  larger  P300s  elicited 
during  the  easier,  first  order  tracking.  Other 
investigators  employing  a  compensatory  tracking 
task  have  also  found  a  systematic  relationship 
between  P300  amplitude  and  system  order  (Wickens, 
Gill,  Kramer,  Ross  and  Donchin,  1981).  These 
studies,  along  with  additive  factors 
investigators  of  manual  control  parameters,  have 
provided  converging  evidence  that  system  order 
has  a  salient  perceptual /central  processing 
component  (Wickens  and  Derrick,  1981;  Wickens, 
Derrick,  Mi  cal  1  i  zi  and  Berringer,  1980)  The 
results  will  also  be  useful  in  the  design  and 
evaluation  of  complex  tracking  tasks.  If 
operators  are  required  to  perform  a  tracking  task 
with  higher  order  system  dynamics  then 
concurrently  performed  tasks  should  be  designed 
so  as  to  minimize  perceptual  /central  processing 
load.  We  see  here,  again,  how  the  ERPs  provide 
data  that  increase  the  theoretical  depth  with 
which  one  can  draw  conclusions  about  the  human 
information  processing  system. 


Figure  9.  Average  parietal  waveforms  elicited  by 
intensifications  of  the  tracking  elements.  The 
left  panel  presents  waveforms  recorded  when  the 
intensity  of  the  target  was  the  relevant  event. 
The  right  panel  displays  waveforms  collected  when 
the  intensity  of  the  cursor  was  relevant.  The 
top  panels  display  waveforms  recorded  during  the 
acquisition  phase,  the  button  panels  present 
waveforms  collected  during  the  alignment  phase  of 
the  target  acquisition  task  (From  Wickens,"  et 
al.,  1982). 


P300  and  Resource  Reci  procity 

The  studies  cited  above  have  demonstrated  a 
robust  relationship  between  P300  amplitude  and 
the  allocation  of  processing  resources  in  a 
secondary  task.  P300s  elicited  by  secondary  task 
probes  decrease  in  amplitude  with  increases  in 
the  perceptual /central  processing  difficulty  of 
primary  tasks.  As  outlined  previously,  one  of 
the  basic  assumptions  of  the  secondary  task 
technique  is  that  increases  in  primary  task 
difficulty  divert  processing  resources  from  the 
secondary  task.  The  decrement  in  secondary  task 
performance  is  believed  to  reflect  this  shift  of 
resources  from  the  secondary  to  the  primary  task. 
Thus,  it  is  assumed  that  there  is  a  reciprocal 
relationship  between  the  resources  allocated  to 
the  primary  and  secondary  tasks.  If  •  this 
assumption  is  correct,  then  it  should  be  possible 
to  demonstrate  that  P300s  .  elicited  by  task 
relevant,  discrete  events  embedded  within  the 
primary  task  are  directly  related  to  primary  task 
difficulty. 

Kramer,  Wickens,  Vanasse,  Heffley  and 
Donchin  (1981)  conducted  an  experiment  in  which 
ERPs  were  elicited  by  task  relevant  events 
embedded  within  a  pursuit  step  tracking  task. 
The  subjects  were  required  to  perform  a  single 
axis  pursuit  step  tracking  task  with  either  first 
order  (velocity)  or  second  order  (acceleration) 
control  dynamics.  In  this  task  the  horizontal 
position  of  a  target  was  determined  by  a  random 
series  of  step  displacements  occurring  at  3  sec 
intervals.  The  subjects  task  was  to  keep  the 
cursor  superimposed  on  the  target.  Difficulty 
was  varied  by  manipulating  two  variables:  the 
degree  of  predictability  of  the  series  of  steps 
and  the  system  order.  In  the  high  predictability 
condition  the  step  changes  alternated  in  a 
regular  right-left  pattern.  In  the  low 
predictability  condition  the  sequence  of  step 
changes  was  random.  The  magnitude  of  the  changes 
was  unpredictable  in  both  conditions.  The  two 
dimensions  of  difficulty,  system  order  and  input 
predictability,  were  crossed  to  create  three 
conditions  of  increasing  difficulty:  first  order 
control  of  predictable  input,  first  order  control 
of  unpredictable  input  and  second  order  control 
of  unpredictable  input. 


Three  different  types  of  probes  were 
employed  as  ERP  eliciting  events.  In  one 
condition  subjects  performed  the  tracking  task 
while  also  counting  the  number  of  occurrences  of 
a  low  pitched  tone  from  a  Bernoulli  series  of 
high  and  low  pitched  tones.  In  the  second 
condition  subjects  counted  the  dimmer  of  two 
flashes  in  a  Bernoulli  sequence.  The  flash 
appeared  as  a  horizontal  bar  along  the  path 
traversed  by  the  target.  In  the  primary  task 
probe  condition,  subjects  counted  the  total 
number  of  step  changes  to  the  left.  Two  control 
conditions  were  also  included:  one  in  which  the 
subjects  counted  the  probes  but  did  not  track  and 
a  second  in  which  subjects  performed  the  tracking 
task  without  counting  the  probes. 
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Figure  10.  Average  Root  Mean  Square  error  and 
subjective  difficulty  ratings  recorded  for  each 
condition  in  a  pursuit  step  tracking  task  (After 
Kramer,  et  al. ,  1981). 

The  important  findings  to  note  in  the  data 
presented  in  Figure  10  are  the  monotonic 
relations  between  the  tracking  difficulty 
manipulations  and  the  subject's  perceived  ratings 
of  difficulty,  as  well  as  those  between  tracking 
difficulty  and  RMS  error.  Thus  both  the 

subjective  and  behavioral  indices  converge  on  the 
same  ordering  of  task  difficulty.  However,  these 
measures  do  not  provide  information  concerning 
the  underlying  resource  structure  of  the  task. 

The  effect  of  tracking  difficulty  on  P300 
amplitude  in  the  auditory  condition  provide 
results  consistent  with  previous  research 
(Isreal,  et  al.,  1980;  Wickens,  et  al.,  1980). 
Thus,  In  the  auditory  condition,  an  increase  in 
the  difficulty  of  the  primary  task  decreased  the 
amplitude  of  the  P300  elicited  by  the  secondary 
task  probes.  In  the  visual  condition  the 
introduction  of  the  tracking  task  resulted  in  a 
reduction  in  the  amplitude  of  the  P300.  However, 
increases  in  tracking  difficulty  failed  to 
produce  any  further  attenuation.  In  the  step 
conditions,  the  amplitude  of  the  P300  elicited  by 
the  discrete  changes  in  the  spatial  position  of 
the  controlled  element  increased  with  increments 
In  the  difficulty  of  the  primary  task.  Thus, -the 
hypothesis  of  resource  reciprocity  between  the 
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Figure  11.  Average  parietal  ERPs  elicited  by 
visual ,  auditory  and  spatial  probes  presented 
concurrently  with  a  pursuit  step  tracking  task  at 
each  level  of  difficulty.  Also  shown  are  the 
ERPs  elicited  during  count-only  control 
conditions  (After  Kramer,  et  al.,  1981). 


primary  and  secondary  tasks  was  confirmed.  One 
final  aspect  of  the  step  tracking  study  has 
considerable  potential  practical  utility.  The 
sensitivity  of  the  P300  elicited  by  visual  steps 
to  resource  allocation  was  observed  independent 
of  whether  or  not  the  subjects  were  required  to 
count  the  stimuli.  These  data  suggest  that 
inferences  from  the  P300  about  resource 
allocation  and  therefore  workload  can  be  made  in 
the  total  absence  of  a  secondary  task 
requirement,  a  considerable  advantage  if  workload 
is  to  be  assessed  unobtrusively  in  real-time 
envi ronments. 


SUMMARY  AND  CONCLUSIONS 


The  investigations  reported  above 
demonstrate  conclusively  that  the  P300  as  a 
secondary  task  can  diagnostically  reflect  primary 
task  workload  variations  of  a 
perceptual /cogniti ve  nature,  uncontaminated  by 
response  factors.  The  absence  of  overt  response 
requirements  provide  it  with  a  considerable 
advantage  over  the  secondary  task,  in  that  the 
oddball  count  task  is  considerably  less 
i  ntrusive. 


As  a  secondary  task  however,  the  probe  task 
is  not  entirely  unobtrusive  and  interpretation  of 
the  measures  still  requires  the  investigator  to 
make  certain  assumptions  about  the  nature  of  the 
primary-secondary  task  interaction  in  order  to 
make  inferences  concerning  operator  workload.  It 
is  for  this  reason  that  our  most  recent 
observations  that  P300  elicited  by  primary  task 
stimuli  also  reflect  resource  allocation  are 
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particularly  encouraging  to  the  utility  of  the 
ERP  as  a  measure  of  workload  in  extra-laboratory 
envi rohments. 

Wereviewed  in  this  chapter,  studies  of  the 
ERP  that  have,  we  think,  one  characteristic  in 
common.  In  each  case  the  ERP  served  as  a  source 
of  information  on  the  timing  or  the  "intensity" 
of  an  information  processing  activity  whose 
behavior  is  not  easily  monitored  by  means  of 
observations  on  overt  responses.  It  would  seem 
that  a  science  of  Human  Engineering  that  is 
interested  in  developing  and  testing  hypotheses 
about  the  internal  structure  and  the  operating 
modes  of  the  human  operator  would  benefit  from 
this  additional  information.  We  advocate  here 
the  use  of  the  ERP  as  an  analytical  tool  that  can 
usefully  aid  in  deepening  our  understanding,  and 
the  measurement  of,  mental  workload. 
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Abstract 


Since  synthetic  speech  stimuli  are  currently  employed  in  commercial  aircraft 
systems,  this  stimulus  type  has  promise  as  a  relatively  unobtrusive  probe  to 
elicit  event-related  brain  potentials  (ERP)  while  people  perform  mental  work 
in  an  operational  environment.  Experimental  work  has  shown  the  feasibility 
of  recording  speech-related  potentials  from  synthetic  speech  stimuli  while 
employing  standard  techniques.  Preliminary  results  have  shown  reliable 
increases  in  the  P300  component  at  the  Pz  site  when  subjects  perform 
mental  work  to  a  word  stimulus  when  compared  to  conditions  where  the  word 
stimulus  is  being  ignored. 

The  ERP  data  has  also  been  processed  for  display  on  a  color  graphics  CRT. 

The  changes  in  scalp  voltage  distributions  show  clearly  differentiated 
patterns  of  activity  as  a  person  senses,  attends,  and  mentally  processes 
the  synthetic  speech  stimuli  we  employed  in  our  experiment.  The  interpre¬ 
tation  of  these  color  patterns  is  based  on  current  understandings  of 
brain  neuroanatomy  and  ERP  theory. 

We  are  studying  the  brain  potential  activity  associated  with  mentally 
processing  language  stimuli  because  it  is  anticipated  that  much  of  the 
mental  workload  of  future  automated  systems  will  be  languaged-based  thinking. 
(For  example;  recalling  procedures  and  system  commands,  formulating  verbal 
communications  and  strategies,  solving  problems,  and  isolating  system 
faults.)  If  we  could  measure  when  the  brain  was  engaged  in  language-based 
processing,  then  we  might  have  a  useful  index  of  mental  workload  which 
could  be  employed  as  a  design  aid  for  complex  automated  systems. 
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I  INTRODUCTION 


A.  Background 

Today's  high-performance  tactical  aircraft  arc  sophisticated,  state- 
of-the-art  weapon  systems  consisting  of  three  components?  the  aircraft, 
the  pilot,  and  the  aircraft's  armament .  Successful  use  of  these  systems 
requires  that  each  component  perform  satisfactorily,  both  individually 
and  in  conjunction  with  each  other.  As  aircraft  and  armament  subsystems 
evolve,  pilot  functions  also  change.  For  example,  as  aircraft  performance 
expands  (greater  speeds,  larger  g-loads),  the  pilot  is  exposed  to  a 
more  stressful  physical  environment .  Pilot  aids  (s,:ch  as  a  reclined 
seat,  antigravity  suit,  and  side-stick  control)  were  developed  to  counter 
this  stressful  environment.  The  pilot's  tsentsl  vc-vl. load  includes  not 
only  flying  the  aircraft,  evaluating  the  tactical  situation  in  combat, 
and  utilizing  the  correct  displays  and  controls,  but.  also  selecting  the 
proper  weapon  from  the  armament  suite.  Many  now  sophisticated  weapons 
are  multimode,  causing  further  stress  on  tl»c  pilot  by  burdening  him 
with  mode  selection.  The  data  and  information  processing  workload  must 
be  divided  between  the  pilot  and  onboard  computers,  and  the  point  at 
which  this  division  is  made  significantly  affects  the  stress  the  pilot 
must  accommodate. 

A  multidimensional  approach  is  necessary  to  measure  pilot  workload, 
stress,  and  performance  quantitatively.  These  tme-ppure meats  require  the 
simultaneous  acquisition  of  aircraft  perform -nee ,  pilot  physiologic 
response,  and  pilot  performance  data  during  tactical  training  flight 
maneuvers.  With  these  data,  the  overall  perform? nee  characteristics  of 
the  weapon  system  can  be  assessed  in  a  reel la  tic  air  combat  situation. 
These  data  may  then  be  used  in  evaluating  weapon  system  design  and  pilot 
training,  in  predicting  pilot  response  to  stressful  environments,  and 
in  developing  countermeasures  to  enhance  pilot;  capabilities. 
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Tools  are  now  available  to  evaluate  both  aircraft  and  pilot  perfor¬ 
mance  individually,  but  they  have  never  been  combined  to  test  the  complete 
weapon  system  (which  includes  the  pilot)  in  a  realistic  air  combat  situa¬ 
tion.  Today's  technology  is  capable  of  evaluating  aircraft/pilot 
performance  in  a  cost-effective  manner.  SRI  is  currently  involved  in 
assessing  all  three  areas  of  aircraft  weapon  system  effectiveness 
(aircraft  performance,  pilot  physiological  response,  and  aircraft  arma¬ 
ment  utilization)  and  is  able  to  correlate  and  analyze  these  assessments 
to  determine  their  unique  relationships. 

B.  Objective 

This  paper  presents  SRI's  current  capability  for  gathering  both 
aircraft  and  armament  performance  data  and  simultaneous  pilot  physio¬ 
logical  response  data.  It  also  explores  the  potential  near-future 
expansion  of  this  capability.  Section  II  describes  SRI's  aircraft  and 
pilot  instrumentation  systems.  Section  III  outlines  current  performance 
measurement  capabilities,  and  Section  IV  describes  the  expanded  measure¬ 
ments  that  will  be  possible  in  the  near  future.  Finally,  Section  V 
summarizes  the  advantages  of  SRI’s  multidimensional  approach. 


II  SYSTEM  DESCRIPTIONS 


A.  TACTS/ACMI 

SRI  is  technical  advisor  to  the  Department  of  Defense  for  develop¬ 
ment  of  an  inflight  aircrew  training  system  called  Tactical  Aircrew 
Combat  Training  System  (TACTS)  by  the  U.S.  Navy  and  Air  Combat  Maneuvering 
Instrumentation  (ACMI)  by  the  U.S.  Air  Force.  The  system  features 
(1)  real-time  tracking,  integration,  processing,  and  display  of  maneuver¬ 
ing  aircraft  and  associated  flight  data,  (2)  computer  simulation  of 
weapon  system  employment  against  which  aircrews  can  exercise  their 
ability  in  flight,  and  (3)  magnetic  recording  of  exercise  data  for 
subsequent  replay  and  in-depth  analysis. 

Today,  five  TACTS/ACMI  systems  (Figure  1)  are  in  operation,  training 
aircrews  in  air-to-air  combat.  The  TACTS/ACMI  arena  for  air-to-air 
combat  training  is  approximately  60  to  80  km  in  diameter  and  15  to  20  km 
in  altitude.  In  this  arena,  opposing  aircrews  in  supersonic  aircraft 
engage  in  free-play  mock  combat  in  an  environment  almost  identical  to 
that  of  real  combat.  Range  instrumentation  tracks  the  aircraft  in  real 
time  in  position,  velocity,  acceleration,  and  attitude;  it  also  provides 
computerized  simulations  of  air-to-air  missiles.  Instrumentation  carried 
in  a  pod  on  each  participating  aircraft  relays  pilot-initiated  missile¬ 
firing  signals  to  ground-based  computers  to  initiate  the  missile  simula¬ 
tions.  Target  tracking  data  recorded  by  the  range  system  are  used  in 
the  simulations  to  determine  whether  the  missile  would  have  resulted  in 
a  kill,  had  a  real  missile  been  fired.  The  results  can  be  immediately 
communicated  to  the  pilot  by  the  Range  Training  Officer  via  ground-to-air 
radio.  Range  activity  is  displayed  in  real  time,  in  three-dimensional 
perspective.  All  data  are  recorded  on  magnetic  tape,  permitting  replay 
during  debriefing  of  all  range  activities. 

In  brief,  TACTS/ACMI  offers  three  generic  features  that  underlie 
both  ^ts  value  as  a  training/test  system  and  its  versatility: 
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FIGURE  1  TACTS/ ACM  I  INSTALLATION  LOCATIONS 


•  Real-time  integration,  processing,  and  display  of  actual  aircraft 
maneuvers  and  flight  data  ("see  it  now"). 

•  Computer  simulations  of  weapon  system  employment  against  which 
aircrews  can  exercise  their  abilities  in  flight. 

•  Magnetic  recording  of  exercise  data  for  subsequent  replay  of 
alphanumeric  and  3-D  graphic  displays  ("see  it  later"). 

Basically,  the  TACTS/ACMI  is  an  integrated  tracking  and  data  trans¬ 
mission  system  consisting  of  four  major  subsystems:  the  Airborne  Instru¬ 
mentation  Subsystem  (AIS),  the  Tracking  Instrumentation  Subsystem  (TIS), 
the  Control  and  Computation  Subsystem  (CCS),  and  the  Display  and  Debrief¬ 
ing  Subsystem  (DDS).  The  system  operates  as  a  closed-loop,  state-vector 
tracking  system  by  combining  in  the  CCS  the  attitude  and  angle-rate 
data  from  the  AIS  with  multilateration  range-tracking  data  from  the  TIS. 
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Airborne  Instrumentation  Subsystem 


2.  Tracking  Instrumentation  Subsystem 

Seven  TIS  ground  stations  and  a  master  station  are  located  to  cover 
the  exercise  arena  geometrically.  The  remote  ground  stations  relay 
master-station  transmissions  to  all  AlS-equipped  aircraft  on  the  range 
and  air-to-ground  transmissions  back  to  the  master  station.  The  master 
station  contains  a  minicomputer  and  associated  peripherals  that  control 
the  TIS  and  process  the  raw  range  data.  A  full-duplex  microwave  data 
link  allows  communication  between  the  TIS  and  the  CCS.  The  master  station 
may  also  be  equipped  with  a  ground-to-air-to-ground  transmitter-receiver, 
so  it  can  operate  as  the  seventh  ground  station. 

The  TIS  incorporates  a  frequency-modulated  continuous  wave  phase- 
comparison  ranging  system  for  range  measurements.  Under  computer  control, 
this  ranging  system  sequentially  interrogates  (Figure  3)  each  participant, 
using  the  remote  sites  to  obtain  multiple  slant-range  measurements  to 
each  aircraft  during  each  interrogation.  Although  only  three  range 
measurements  are  required  to  determine  position,  the  system  attempts  to 
collect  all  seven  ranges  to  provide  an  overdetermined  solution  and  to 
minimize  data  loss  due  to  shadowing  and  aircraft  blockage  during  maneuvers. 
The  TIS  relays  raw  range  data  and  the  downlink  data  messages  to  the  CCS. 


AIS 


FIGURE  3  AIS  POD  INTERROGATION  AND  RE^ONSE  SEQUENCES 
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3. 


Control  and  Computation  Subsystem 


The  CCS,  the  "brain"  of  the  TACTS/ACHI,  consists  of  a  large-capacity 
multiprocessor  computational  system  and  a  microwave  data  link  for  communi¬ 
cating  with  the  local  and  remote  DDSs  and  with  the  TIS  master  station. 
Current  systems  use  a  multiprocessor  configuration  with  shared  memories. 

A  complex  filtering  technique  is  used  to  integrate  the  TIS-collected 
ranging  and  inertial  data  on  up  to  8  high-dynamic  aircraft  and  ,to  provide 
real-time  state-vector  tracking  data  to  the  DDS  for  display  and  recording. 
Ranging  data  are  used  in  a  separate  filter  for  position-only  tracking 
of  12  additional  aircraft.  Tracking  results  are  available  for  display 
at  the  DDS  within  0.2  s  of  real  time.  Using  both  the  state-vector  track¬ 
ing  information  processed  through  the  TIS  and  CCS  computers  and  the 
timing  of  the  weapons  firing  signals,  the  CCS  calculates  the  miss  distance 
between  the  simulated  weapon  and  the  target  and  scores  the  pilot  accord¬ 
ingly. 

Tracking  data  are  also  provided  to  a  CCS  subprogram  that  calculates 
flight  safety.  Aircraft  are  monitored  for  hazardous  performance  and 
for  location  within  the  range  boundary.  Appropriate  alarms  are  trans¬ 
mitted  to  the  RTO  at  the  DDS  when  any  selectable  preset  limit  is  violated. 

To  process  an  exercise,  the  CCS  accepts  exercise  data  for  setup 
and  termination  from  the  DDS  and  responds  to  DDS-supplied  fighter/target 
designations,  hazard  limit  changes,  and  other  exercise  data.  To  sequence 
its  programs  correctly  and  to  monitor  pilot  performance  and  hazards, 
the  CCS  compares  fighter  and  target  positions,  velocities,  and  accelera¬ 
tions  against  predetermined  parameters.  In  addition,  the  CCS  maintains 
a  statistical  summary  of  results  of  all  weapons  used  during  each  mission 
based  on  the  weapon's  launch-boundary  compliance  and  kill  determination 
data.  The  CCS  performs  real-time  operability  test  and  calibration  and 
accepts  status  messages  from  the  TIS  and  DDS,  as  well  as  from  its  own 
computer  and  peripheral  equipment.  These  status  data  are  included  in 
the  summary  maintained  by  the  CCS  and  communicated  to  the  DDS. 
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4.  Display  and  Debriefing  Subsystem 

Weapons  results,  aircraft  maneuvers,  and  interaircraft  parameters 
are  graphically  displayed  in  real  time  at  the  DDS  facility  (Figure  4) 
and  are  recorded  for  replay  during  postmission  debriefing.  Aircrews 
aloft  also  receive  auditory  notification  of  weapons  results  in  real 
time.  This  capability  teaches  aircrews  to  recognize  weapons-envelope 
boundaries  and  other  parameters  necessary  for  successful  weapon  delivery. 


LARGE-SCREEN 
GRAPHICS  DISPLAY 


LARGE-SCREEN 

ALPHANUMERICS 

DISPLAY 


CONTROLS 

FIGURE  4  DDS  DUAL-CONTROL  DISPLAY  CONSOLE 


The  DDS  graphic,  three-dimensional  representation  of  the  range  and 
exercise  activity  uses  perspective,  varies  size  with  distance,  and  reduces 
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intensity  with  distance.  Range  terrain  and  aircraft  locations,  altitudes, 
attitudes,  and  flight  paths  are  displayed  on  both  situation-type  CRT 
displays  and  large  color  observation  screens.  These  situation  displays 
have  azimuth  and  elevation  coordinate  rotation,  automatic  centering  of 
the  exercise  centroid,  and  "zoom"  capabilities.  Missions  can  thus  be 
viewed  from  virtually  any  desired  aspect  in  real  time  or  during  playback. 
Figure  5  shows  examples  of  DDS  situation  displays.  Maneuvering  aircraft 
tracks  are  presented  as  variable  moving  images  with  selectable  time- 
history  ribbons.  Aircraft  ground  tracks  and  range  topography  representa¬ 
tion  augment  the  three-dimensional  character  of  the  display.  In  addition, 
a  pilot's  view  (Figure  5d)  may  be  selected,  which  pictures  the  engagement 
from  the  cockpit  of  any  selected  aircraft  on  the  range. 

Dynamic  flight  data  are  displayed  alphanumerical ly  (Figure  6). 
Indicated  airspeed,  altitude,  angle  of  attack  and  normal  acceleration 
(g),  as  well  as  flight  relationships  between  aircraft  and  target  (range, 
bearing,  closing  velocity,  angle  of  the  tail,  and  antenna  train  angle), 
are  typical  of  the  information  available  in  real  time.  Summary  data 
are  automatically  displayed  and  printed  at  the  end  of  each  mission. 

A  selectable  engineering  data  display  (Figure  7)  provides  a  wide 
range  of  engineering  parameters,  including  the  range  position,  velocity, 
acceleration,  and  attitude  of  each  participant. 

During  replay,  unlimited  variations  of  the  three-dimensional  display 
can  be  selected  for  analysis,  regardless  of  what  was  displayed  during 
live  monitoring.  The  operator  has  complete  freedom  to  start,  stop, 
change  scale,  and  rotate  coordinates.  In  addition,  hard  copies  of  any 
data  or  display  can  be  printed  for  in-depth  analysis.  A  more  detailed 
technical  description  of  TACTS  is  given  by  McHenry  and  York.^ 

B.  SRI's  Ambulatory  Physiologic  Monitoring  System  (APMS) 

SRI  has  also  specialized  in  developing  advanced  physiological  monitor¬ 
ing  systems,  particularly  for  use  in  an  ambulatory  environment,  for 
many  years.  Typical  projects  include: 
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(a)  PLAN  VIEW  (40-nini  scale) 


(b)  SLIGHT  ELEVATION  (10-nmi  scale) . 


(c)  MISSILE  FIRE  (2.5-nml  scale)  (d)  PILOT  VIEW  (with  reasons  for  miss) 

FIGURE  5  DOS  SITUATION  DISPLAYS 
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FIGURE  6  FLIGHT  DATA  DISPLAY 


FIGURE  7  DDS  ENGINEERING  DATA  DISPLAY 


•  Ambulatory  blood  pressure  monitor 

•  Ambulatory,  8-channel,  12-hour  EEG  recording  system 

•  NASA  Space  Shuttle  in-flight  physiological  monitoring  system 

•  Stress  test  blood  pressure  monitor 

•  NASA  venous  occlusion  cuff  and  controller. 

Many  of  the  ihstruments  developed  by  SRI  for  NASA  have  been  for 
use  aboard  the  Spate  Shuttle.  These  instruments  are  built  with  the 
quality  control  and  environmental  testing  required  for  flight  certification. 

Several  systems  have  been  combined  to  form  the  Ambulatory  Physiologic 
Monitoring  System  (APMS),  the  elements  of  which  are  described  below. 

1 .  Recording  System 

A  key  element  in  many  of  SRI's  ambulatory  monitoring  systems  is 
the  8-channel  analog  or  digital  cassette  recording  system.  This  record¬ 
ing  system  has  been  developed  over  the  past  nine  years  under  the  auspices 
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of  (first)  the  National  Institutes  of  Health,  for  long-term  EGG  recording, 
and  (presently)  NASA,  as  a  general-purpose  data  recorder.  The  system, 
which  evolved  through  several  versions  of  both  recorder  and  playback 
systems,  is  now  fully  operational  and  has  been  certified  for  NASA  use 
in  Space  Shuttle  programs. 

The  current  recorder  system  consists  of  a  battery-powered,  single¬ 
unit,  digital  cassette  recorder  (Figure  8)  and  a  desk-top,  microprocessor- 
based,  data  playback  system  (Figure  9).  The  recorder  has  eight  indepen¬ 
dent  data  channels.  These  channels  can  be  configured  to  be  all  digital, 
or  six  digital  and  two  analog  channels.  The  digital  channels  are 
digitized  to  10  bits,  giving  a  0.12  accuracy  and  60-dB  dynamic  range. 

The  analog  channels,  if  used,  have  a  22  accuracy  and  40-dB  dynamic  range. 
The  analog  channels  are  capable,  however,  of  a  higher  frequency  response 
than  the  digital  channels.  The  recorder  can  be  configured  for  three 
different  recording  speeds,  allowing  recording  time  to  vary  between  2 
and  10  hours.  Available  bandwidths  and  recording  times  are  given  in 
Table  1.  A  time  code  can  be  inserted  once  a  minute  on  the  digital  chan¬ 
nels,  does  not  require  a  separate  data  channel,  and  can  be  used  to 
synchronize  the  recorder  to  the  TACTS  system.  The  recorder  gains  in 
flexibility,  small  size,  and  low  power  consumption  from  the  use  of  comple¬ 
mentary  metal-oxide-semiconductor  (CMOS)  microprocessor  technology  packaged 
in  custom  hybrid  modules. 

2.  Playback  System 

A  playback  system  is  used  to  recover  the  data  from  the  cassette 
recorder.  Data  are  played  back  faster  than  real  time  and  are  transferred 
directly  to  a  medium-sized  computer.  To  do  this,  a  bipolar  bit  slice 
microprocessor  controls  data  playback  and  data  transfer.  Once  transferred 
and  stored,  the  data  can  be  examined  and  processed  at  will  using  the 
capabilities  of  the  host  computer. 

The  primary  advantages  of  this  recorder  system  are  that:  (1)  eight 
channels  of  data  can  be  Recorded,  (2)  digital  recording  provides  signal 
fidelity  equivalent  to  laboratory  conditions  (accuracy  of  0.12), 
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DIGITAL  WATCH 


FIGURE  8  EXISTING  8-CHANNEL  EEG  RECORDER,  ELECTRONICS,  AND 
BATTERY  PACK 

(3)  long-term  recording  is  available,  (4)  the  recorder  is  small  in  size, 
and  (5)  data  playback  and  processing  is  convenient  and  flexible. 

3.  Ambulatory  Blood  Pressure  System 

For  SRI's  APMS,  the  cassette  data  recorder  is  combined  with  cuff 
inflation  and  control  and  with  appropriate  sensors  and  signal  conditioning 
to  create  a  system  that  records  all  the  data  necessary  to  determine 
blood  pressure,  even  when  the  subject  is  moving.  A  motor  and  air  pump 
combination  inflates  the  cuff.  Two  miniature  air  valves  under  micropro¬ 
cessor  control  gradually  deflate  and  empty  the  cuff  as  desired.  Two 
pressure  transducers  provide  backup  system  control  and  cuff  pressure 
data.  A  microphone  is  used  to  obtain  Korotkov  sounds,  and  ECG  electrodes 
are  used  to  obtain  high-quality  ECG  data  for  recording.  The  system  is 
powered  from  rechargeable  nickel-cadmium  batteries.  In  addition  to  the 
armcuff,  the  system  is  configured  in  two  belt-worn  packages,  one  contain¬ 
ing  the  battery  pack,  motor,  air  pump,  and  pneumatics,  and  the  other 
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FIGURE  9  EXISTING  PLAYBACK  SYSTEM  (block  diagram) 


Table  1 

PROPOSED  CASSETTE  DATA  RECORDER  PERFORMANCE  CHARACTERISTICS 


Digital 

Sample 

Rate 

(samples/s/ 

channel) 

Digital 
Channe 1 
Encoded 
Analog 
Bandwidth 
(Hz) 

Digital 

Data 

(bits/s/ 

channel) 

Data 

Density 

(bits/in./ 

track) 

Tape  Record 
Time 
(h) 

Analog 

Bandwidth 

per 

Channe 1 
(kHz) 

Analog 

Dynamic 

Range 

(dB) 

Dec. 

Free. 

C-90 

C-120 

0.1875 

3/16 

100 

40 

1100 

5866 

8 

10.6 

1.5 

30 

0.375 

3/8 

200 

80 

2200 

5866 

4 

5.3 

3.0 

35 

0.75 

3/4 

400 

160 

4400 

5866 

2 

2.6 

6.0 

40 
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containing  the  recorder,  cuff  control  system,  and  user  interface  and 
controls . 

This  ambulatory  blood  pressure  system  features  (1)  preprogrammed 
timed  intervals  between  inflations,  (2)  preprogrammed  pressure  limits, 

(3)  user-initiated  inflation  or  deflation  override,  (4)  deflation  rate 
based  on  heart  rate,  (5)  user  display  showing  cuff  pressure  during  infla¬ 
tion,  and  (6)  numerous  backup  safety  provisions  to  protect  against  over¬ 
inflation  and  excessive-time  inflation. 

The  system  can  inflate  to  pressures  up  to  300  mmHg  and  can  run 
continuously  for  4  hours  with  up  to  100  inflations.  Longer  monitoring 
times  are  available  if  intermittent  recording  is  provided. 

Blood  pressure  data  are  obtained  after  the  data  cassette  is  retrieved 
and  played  back  through  the  playback  system.  A  highly  sophisticated 
digital  filtering  and  processing  algorithm  is  used  to  examine  the  recovered 
Korotkov  sounds,  ECG  signal,  and  cuff  pressure  to  determine  blood  pressure. 
This  algorithm  can  obtain  accurate  blood  pressures  even  in  the  presence 
of  high  external  noise  levels  and  subject  motion  artifacts.  In  fact, 
this  system  has  been  used  successfully  in  obtaining  automated  blood 
pressure  measurements  during  both  bicycle  and  treadmill  stress  testing. 

4.  Antigravity  Suit  Evaluation  Instrumentation 

A  third  system,  the  antigravity  suit  in-flight  evaluation  instrument¬ 
ation,  developed  by  SRI  for  use  by  NASA  on  the  Space  Shuttle,  combines 
the  cassette  data  recorder  and  the  blood  pressure  measuring  system  with 
Doppler  blood  flow  instrumentation,  phonocardiogram,  and  g-suit  pressure 
monitors  to  form  a  highly  sophisticated  monitoring  system.  The  system 
is  battery-powered  and  self-contained  in  a  single  package  that  attaches 
to  the  seat  of  the  individual  being  monitored.  All  data,  including  the 
raw  data  to  determine  Doppler  blood  flow,  are  recorded.  During  data 
playback,  the  various  blood  pressure  and  blood  flow  data  are  processed 
and  recovered,  along  with  the  ECG,  phonocardiogram,  and  antigravity 
suit  pressure  waveforms. 
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SRI  has  also  built  the  hardware  necessary  to  measure  body  temperature 
and  respiration,  but  has  not  yet  integrated  it  into  the  APMS.  Temperature 
is  measured  with  a  small  semiconductor  temperature  probe;  respiration 
is  measured  with  SRI's  piezoelectric  chest  expansion  strain  guage.  These 
potential  additions  to  the  APMS  are  described  in  Section  IV. 
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Ill  CURRENT  DATA  MEASUREMENT  CAPABILITIES 


Pilot  performance  depends  on  optimal  weapons  selection,  evaluation 
of  the  tactical  situation,  and  accurate  targeting,  as  well  as  on  aircraft 
and  weapon  performance.  Likewise,  the  overfill  effectiveness  of  the 
aircraft  and  the  weapon  system  depends  on  the  pilot's  ability  to  operate 
them.  We  therefore  need  to  determine  the  optimal  level  of  stress  at 
which  the  pilot  is  most  alert,  yet  able  to  accotmnodate  the  many  decisions 
facing  him.  This  section  describes  SRI's  current  capabilities  for  measur¬ 
ing  both  aircraft  performance  and  pilot  stress  and  workload. 

A.  Aircraft  Data 

The  TACTS/ACMI  system  provides  complete  aircraft  state-vector  track¬ 
ing  data,  including  load  factors,  velocities,  position,  angular  rates, 
attitudes,  and  air  data  sensor  information.  Table  2  lists  the  specific 
data  available  from  the  TACTS/ACMI  system.  Examples  of  these  data  are 
presented  graphically  in  Figures  10-12,  which  show  both  TACTS/ACMI  and 
onboard  reference  instrumentation  data.  The  TACTS/ACMI  data  were  taken 
directly  from  CCS  range  tapes.  In  all  cases,  the  range  data  compare 
closely  to  the  aircraft  reference  data.  Figure  10  shows  examples  of 
state-vector  tracking  data.  Figure  11  shows  examples  of  air  data  sensor 
information  available  from  the  range.  Finally,  Figure  12  is  an  example 
of  a  parameter  derived  from  the  data  provided  by  the  system. 

B.  Pilot  Physiologic  Data 

The  current  SRI  APMS  can  be  used  conveniently  to  obtain  and  record 
biologic  signals  of  respiration,  heart  rate,  and  blood  pressure  from  a 
subject  under  operational  conditions.  To  obtain  these  measures,  several 
transducers  are  placed  on  the  test  subject:  one  ground  electrode,  two 
ECG  electrodes,  one  solid-state  chest  strain  gauge,  and  a  standard  blood 
pressure  armcuff.  A  small  microphone  embedded  in  the  armcuff  records 
the  Korotkov  sounds  during  the  periodic  blood  pressure  measurements. 
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Table  2 


TACTS /ACMI  DATA 


State  Vector  Tracking  Data 

Position  (x,  y,  z) 
Velocity  (x,  y,  z) 
Acceleration  (x,  y,  z) 
Attitude  (wx,  Wy,  wz) 

Derived  Data 

Angular  acceleration 
We igh  t 

Altitude  (above  terrain) 


CAPABILITY 


Air  Data  Sensor  Information 

Dynamic  pressure 
Static  pressure 
Mach  number 
Calibrated  airspeed 
Angle  of  attack 
Angle  of  sideslip 


3646  3660  3662  3664  3666  3668  3860  3862 


TIME  —  ■  X  10 


FIGURE  12  DERIVED  DATA  (roll  acceleration) 


1.  Respiration 

Respiratory  signals  can  be  acquired  by  a  variety  of  transducers. 

The  APMS  uses  an  SRI-developed  piezoelectric  chest-wall  strain  gauge  to 
monitor  respiration  without  the  restrictions  of  ordinary  gauges.  Mercury- 
tube  strain  gauges,  for  example,  though  commonly  used  for  sensing  chest 
and  abdomen  expansion,  require  a  sophisticated  amplifier.  A  more 
important  disadvantage  is  that  mercury  is  quite  toxic,  and  the  risk  of 
a  broken  gauge  is  great.  Impedance  pneumography  is  subject  to  body 
movement  and  heart  motion  artifact.  Pneumotachography  gives  good  data, 
but  restricts  the  subject's  movements. 

The  SRI  strain  gauge  has  none  of  these  disadvantages.  Figure  13 
shows  a  prototype  version.  The  transducer  is  a  piezoelectric  crystal 
about  the  size  of  a  matchhead.  Connections  to  the  crystal  and  its  rubber 
housing  combine  to  produce  a  package  about  the  size  of  the  eraser  on 
the  end  of  a  pencil.  The  preamplifier  for  the  unit  requires  only  two 
integrated  circuits.  The  output  from  the  preamplifier  is  a  high-level 
(i.e.,  hundreds  of  millivolts)  voltage  directly  proportional  to  the 
strain  exerted  on  the  transducer. 

In  operation,  the  transducer  is  placed  in  the  middle  of  the  back. 

Its  strings  are  cinched  lightly  around  the  torso  with  a  double  D-ring. 

The  cinching  should  be  tight  enough  to  ensure  that  the  transducer  is 
always  in  tension,  but  not  tight  enough  to  restrict  breathing.  A  little 
elasticity  in  the  strings  ensures  the  latter  condition. 

Figure  14  shows  the  output  of  the  SRI  strain  gauge  compared  with  a 
reference  mercury-filled  strain  gauge.  For  this  comparison,  the  SRI 
gauge  girdled  the  entire  abdomen,  while  the  mercury  device  was  sensing 
motion  only  on  one  side  of  the  abdomen.  The  device  provides  respiration 
signals  independent  of  body  position  and  reasonably  free  of  motion 
artifact. 


2.  Heart  Rate 

SRl's  ambulatory  recorder  systems  can  record  ECG  signals  accurately 
(to  0.1Z).  Upon  data  playback,  this  ECG  signal  can  be  analyzed  to 
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FIGURE  13  PROTOTYPE  SRI  STRAIN  GAUGE 
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determine  R-to-R  intervals,  average  heart  beat,  and  (because  of  the 
recording  fidelity)  S-T  segment  depression.  Other  ECG  data  analysis 
techniques  could  also  be  used,  if  desired. 

3.  Blood  Pressure 

Of  the  three  physiologic  parameters  measured  by  the  APMS,  blood 
pressure  is  by  far  the  most  difficult  to  obtain.  SRI  has  designed  and 
constructed  several  ambulatory  blood  pressure  measuring  systems  and 
currently  has  an  operating  prototype  of  the  APMS  system  described  in 
this  paper. 

After  the  data  have  been  obtained  using  the  on-body  portion  of  the 
APMS  (Section  II),  the  recorded  data  are  played  back  through  the 
cassette  data-playback  system  and  transferred  to  a  computer.  The 
computer  is  then  used  to  analyze  the  data  and  to  determine  systolic  and 
diastolic  blood  pressure. 

The  software  package  uses  the  intervals  between  the  ECG  R-wave 
peaks  and  the  onset  of  the  Korotkov  sounds  (the  R-K  interval)  during  a 
cuff  deflation  to  determine  the  systolic  and  diastolic  blood  pressures 
each  time  the  blood  pressure  cuff  is  inflated.  A  sophisticated 
computer  algorithm  is  used  to  determine  systolic  and  diastolic  blood 
pressures.  This  algorithm  (a  detailed  description  of  which  is  beyond 
the  scope  of  this  paper)  makes  it  possible  to  obtain  accurate  blood 
pressure  readings  in  the  presence  of  external  noise  and  subject 
movement  (even  strenuous  physical  activity)  that  normally  would  mask 
the  Korotkov  sounds  and  make  accurate  blood  pressure  readings 
impossible. 

C.  Pilot  Tasking  Data 

As  part  of  the  TACTS/ACMI  system,  all  weapons  selection  data  are 
recorded  during  real-time  combat  exercises  and  are  available  for  later 
replay  to  determine  whether  the  pilot  best  utilized  the  armament  avail¬ 
able.  Weapons  data  currently  recorded  on  the  TACTS/ACMI  system 
include:  bombs,  mines,  Falcon  (AIM-4),  Sparrow  (AIM-7),  Shrike 
(AIM-45),  and  Sidewinder  (AIM-9) .  The  individual  signals  available 
from  each  weapon  are  listed  in  Table  3. 
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Table  3 


INDIVIDUAL  SIGNALS  ASSOCIATED  WITH  SPECIFIC  WEAPON  TYPE 


Bombs 


Falcon  (AIM- 4) 


Trigger 

Mines 


Trigger-arm 

Heat 

Radar 


Trigger 

Shrike  (AIM-54) 

Fire  ready 
Fire 

Direction  finding  angles 
Pulse  repetition  frequency 

Sidewinder  (AIM- 9) 

Coolant 
Seeker  angles 
Trigger-arm 
Audio 


Gun  select 
Trigger-safe 

Sparrow  (AIM- 7) 

Head  aim  angles 

Range 

Trigger 

Gate  select 

CW  power 

Dogfight 


f 
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IV  FUTURE  DATA  MEASUREMENT  CAPABILITIES 


A.  Aircraft  Data 

Plans  are  underway  for  an  extensive  TACTS/ACMI  evaluation  of  both 
the  F-16  and  F/A-18  aircraft.  For  these  operations,  the  TACTS/ACMI  is 
being  adapted  to  collect  aircraft  performance  data  from  the  data  multi¬ 
plex  bus  in  these  new  aircraft.  The  additional  data  that  will  be 
available  will  expand  the  TACTS/ACMI  utility  for  both  the  training  and 
test  communities  of  users.  The  TACTS/ACMI  system  will  be  able  to  access 
any  aircraft  data  that  are  available  on  the  avionics  subsystem.  For 
example,  if  engine  performance  data  are  required,  the  avionics  can  supply 
rpm,  inlet  temperature,  outlet  temperature,  fuel  flow,  and  power  lever 
angle.  If  specific  data  are  needed  to  analyze  flight  dynamics,  the 
serial  data  capability  can  supply  flight  control  surface  position  and 
rates.  The  future  data  accessability  of  the  TACTS/ACMI  system  will 
increase  significantly  as  new  and  modified  aircraft  begin  training  with 
this  system. 

B.  Pilot  Physiological  Data 

SRI  plans  to  expand  the  present  APMS  capabilities  to  measure  skin 
or  rectal  temperature,  temporal  blood  flow,  peripheral  blood  flow  and 
pooling,  and  cardiac  output.  The  technology  required  for  the  ambulatory 
measurement  of  these  additional  parameters  is  fully  developed  and  has 
been  demonstrated  on  centrifuge  tests  at  NASA-Ames  Research  Center. 

1.  Temperature 

SRI  uses  a  semiconductor  transducer  to  measure  body  temperature. 

This  temperature-measuring  circuit  may ( be  used  to  obtain  skin,  rectal, 
or  tympanic  membrane  temperature. 
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2.  Temporal  Blood  Flow 

In  the  past,  human  +GZ  acceleration  tolerance  has  partly  relied  on 
subjective  criteria  related  to  peripheral  loss  of  vision.  A  noninvasive 
Doppler  flow  system  has  recently  been  developed  by  Dr.  Salvadore  A.  Rositano 
at  NASA-Ames  Research  Center.  This  system  may  be  used  to  obtain  an 
objective  end-point  criteria  for  +GZ  tolerance.  It  consists  of  miniature 
8-MHz  Doppler  sensors  placed  on  the  forehead  over  both  frontal  branches 
of  the  temporal  arteries  to  detect  blood  flow  to  the  head  from  backscattered 
ultrasound.  As  reported  by  Dr.  Rositano,! 

Over  100  subjects  have  now  been  studied  during  more 
than  2,000  centrifuge  runs.  Objective  changes  in 
temporal  artery  flow  velocity  consistently  preceded 
visual  degradation  for  each  subject  during  all  acceler¬ 
ation  profiles.  No  subject  has  gone  unconscious 
without  first  exhibiting  a  minimum  6  seconds  of 
total  flow  cessation.  Retrograde  flow  followed  by 
complete  flow  cessation  always  preceded  central 
light  loss. 

Results  indicate  that  this  method  can  be  successfully 
used  with  a  wide  variety  of  tasks  during  exposure 
to  +GZ  acceleration.  It  is  recommended  for  use 
during  evaluation  of  protective  maneuvers  or  devices 
on  the  centrifuge  or  during  actual  flight  in  high 
performance  aircraft. 

A  typical  data  trace  obtained  during  Rapid  Onset  Run  (ROR)  +Gz 
centrifugation  is  shown  in  Figure  15.  As  stated  in  this  report, 

When  blackout  was  approached  (range  2.7  to  4.6  G), 
eye  level  arterial  blood  pressure  (Pa)  began  to 
fall  c one omni t ant  with  the  occurrence  of  retrograde 
flow  in  the  temporal  artery.  Zero  forward  temporal 
artery  flow  (Qta)  was  determined  by  both  graphic 
and  audio  recordings  six  seconds  (range  4-9  s)  prior 
to  blackout.  Eye  level  mean  arterial  pressure  (Pa) 
decreased  to  20  +1  mmHg  (n  *  16)  when  zero  forward 
Qta  was  initially  recorded.  Arterial  pressure  and 
temporal  artery  flow  velocity  increased  simultaneously 
during  centrifuge  deceleration,  with  a  significant 
increase  in  pressure  and  flow  velocity  occurring 
post  run  when  compared  to  previous  values. 

The  point  of  blackout  is  denoted  by  the  vertical  line  in  Figure  15, 
where  the  subject's  central  light  loss  (CLL)  occurred. 
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FIGURE  15  EFFECT  OF  RAPID  ONSET  ACCELERATION  ON  EYE-LEVEL  BLOOD  PRESSURE 
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Rositano's  system  is  currently  being  modified  for  use  during  Space 
Shuttle  flights.  It  will  be  incorporated  into  a  flight  instrument  package 
being  built  by  SRI  for  NASA-Johnson  Space  Center.  The  output  signals 
from  the  Doppler  sensors  will  be  recorded  on  the  SRI  cassette  tape  recorder 
as  used  in  the  APMS. 

3.  Peripheral  Blood  Flow 

Peripheral  blood  flow  and  pooling  have  also  been  measured  during 
+GZ  centrifugation  using  impedance  plethysmography.  This  work  was  done 
during  the  past  five  years  as  part  of  male/female  bed-rest  studies  con¬ 
ducted  by  NASA  at  the  Ames  Research  Center.  This  technique  was  used  to 
quantify  the  hemodynamic  responses  of  45-  to  65-year-old  subjects  to 
Space  Shuttle  reentry  +GZ  acceleration  before  and  after  a  10-day  simulated 
exposure  to  weightlessness.  These  measures  were  also  used  to  assess 
the  effectiveness  of  wearing  an  inflated  antigravity  flight  suit  during 
simulated  reentry  acceleration. 

Each  subject  was  instrumented  as  shown  in  Figure  16.  The  impedance 
plethysmograph  introduced  a  high-frequency  (about  50  kHz),  low-amperage 
(about  0.1  mA  rms)  electrical  current  between  shoulder  and  foot  disposable 
EKG  electrodes  (ij  and  I2) •  Simultaneous  measurements  of  baseline 
resistance  (R0)  and  pulse  volume  resistance  changes  (  R)  were  made  in 
the  leg  between  similar  ankle  and  upper-thigh  recording  electrodes. 

Records  of  R0  were  analyzed  according  to  Nyboer^  to  determine  the  amount 
of  blood  accumulated  in  the  leg  during  each  test  sequence.  Leg  blood 
flow  indices  were  calculated  using  the  segmental  R  values. 3 

The  acceleration  tests  took  place  during  bedrest  studies  that  con¬ 
sisted  of  a  9-day  control  period,  10  days  of  absolute  bed  rest,  and 
5  days  of  recovery.  Each  subject  was  exposed  to  +GZ  centrifugation  six 
times  during  the  study — on  days  1,  2,  and  7  of  the  control  period  to 
establish  a  baseline  for  acceleration  tolerance,  and  on  bed-rest  days  7 
through  9  to  determine  the  incidence  of  deconditioning.  During  all 
centrifugation  exposures,  the  subjects  were  tested  both  with  and  without 
inflated  antigravity  suits  to  evaluate. the  effectiveness  of  this 
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FIGURE  16  INSTRUMENTED  SUBJECT  DURING  +Gz  CENTRIFUGE  TESTS 

countermeasure.  On  days  1  and  2  of  the  control  period  and  days  7  and  8 
of  bed  rest,  the  subjects  were  exposed  to  acceleration  profiles  of  1.5  +GZ 
and  2.0  +GZ — the  levels  projected  for  Space  Shuttle  Reentry  (see  Figure  17). 
Each  of  these  tests  included  10  min  in  the  upright  position  to  simulate 
1.0  +GZ  during  landing  following  the  simulated  reentry  acceleration 
stress.  On  day  3  of  the  control  period  and  on  day  9  of  bed  rest,  the 
subjects  underwent  a  +3.0  Gz  acceleration  exposure  as  shown  in  Figure  17 
for  15  min  as  a  general  test  of  acceleration  tolerance.  During  each 
suited  test,  the  antigravity  suits  were  inflated  1.0  psia/g. 

Figure  18  shows  a  typical  data  trace  obtained  during  one  of  the 
+3.0  Gz  tests  before  bed  rest.  The  antigravity  suit  was  not  inflated 
during  this  test  sequence.  The  top  trace  in  each  portion  of  Figure  18 
shows  the  subject's  ECG  signal.  The  bottom  trace  in  each  segment  shows 
the  subject 's  leg  blood  flow  pulse.  Figure  18a  was  taken  at  2-min 
elapsed  time  during  the  resting  control  period  preceding  centrifugation. 
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ACCELERATION- — g  ACCELERATION  —  g  ACCELERATION 


FIGURE  17  ACCELERATION  PROFILES  USED  DURING  CENTRIFUGE  TESTS 
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+3.0  G2  —  PRE 


FIGURE  18  IMPEDANCE  DATA  TRACE  FROM  CENTRIFUGE  TESTS 
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Figure  18b  shows  the  same  data  traces  obtained  at  the  time  that  the 
subject  initially  reached  +3.0  Gz  acceleration  at  9-min  elapsed  time. 

Figure  18c  shows  the  traces  obtained  after  9-min  exposure  to  +3.0  Gz 
(at  18-min  elapsed  time). 

By  comparing  the  pulse  amplitudes  the  R  traces  at  the  various 
times  (use  the  X100  sensitivity  at  18  min  in  Figure  18),  one  can  deter¬ 
mine  the  extent  of  decreased  perfusion  that  is  produced  by  increased 
+GZ  stress. 

The  grouped  mean  (+S.E.),  percent  leg  pooling  (%  DVE)  and  leg  blood 
flow  (X  LBF)  values  obtained  during  the  1.5  +GZ  pre-bed-rest  tests  of 
45-  to  55-year-old  men  are  shown  in  Figure  19.  These  data  illustrate 
the  hemodynamic  responses  to  the  Space  Shuttle  reentry  profile  (Figure  19a) 
and  show  how  these  measures  are  altered  by  inflation  of  the  antigravity 
flight  suit.  The  asterisks  in  Figure  19  denote  significant  (P  0.05) 
changes  that  occurred  for  each  parameter  between  the  various  important 
events/times  during  the  +GZ  profile.  The  plus  symbols  show  significant 
(P  0.05)  differences  between  the  inflated  (dashed  lines)  and  the  non- 
inf  lated  (solid  lines)  tests  at  specific  times. 

These  results  demonstrate  that  impedance  plethysmography  can  be 
used  to  quantify  hemodynamic  responses  of  flight-suited  subjects  to 
periods  of  dynamic  stre'ss.  These  post  studies  used  a  rack-mounted 
impedance  unit.  This  Equipment  could  be  miniaturized,  however,  and 
incorporated  into  the  APMS  instrumentation  for  ambulatory  measurements. 

4.  Cardiac  Output 

Impedance  plethysmography  has  also  been  used  by  other to  measure 
cardiac  output  and  stroke  volume  during  exercise.  An  ambulatory  impedance 
instrument  would  extend  this  capability  into  the  high-performance  aircraft 
environment . 

C.  Pilot  Tasking  Data 

Table  4  lists  the  cockpit  controls  and  displays  that  will  be  moni¬ 
tored  when  the  TACTS/ACMI  system  is  used  for  F/A-18  training.  Table  4 
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%  LBF  —  ml/100  ml/min  %  DVE 


does  not  list  each  discrete  signal  associated  with  a  given  display, 
control,  or  weapon,  but  it  does  give  the  general  description  of  its 
function.  Each  function  normally  has  several  selectable  discretes 
associated  with  it.  When  the  system  becomes  operational,  each  discrete 
will  be  available  through  the  TACTS/ACMI  system.  The  system  will  be 
capable  of  duplicating  in  the  replay  mode  all  tasks  associated  with 
display  utilization,  control  inputs,  and  weapon  selection. 

Table  4 

F/A-18  COCKPIT  CONTROLS  AND  DISPLAYS 


Head-Up  Display 

Basic  flight  data 
Flight  path/pitch  ladder 
Waterline  symbol 
Velocity  vector  caging 
Clutter  reject 
Landing  and  steering  data 
Advisory  data 

Flight  Control  Stick 
and  Throttle  Switches 

A/A  weapon  select 
A/G  weapon  release 
Gun  missile  trigger 
Flight  control 
Display  select 
Radar  mode  select 
Communication  select 
EW  select 


Air-to-Ground  Mode 

Conventional  bombs 
Laser-guided  bombs 
Rockets 

Maverick  (AGM-65E) 

HARM  (AGM-88A) 

Gun 

Electronic  Warfare  Displays 

Chaff 

Flare 

Jammer 

Dispenser 

ECM 

ECCM 

FLIR 

LST 

Radar  mode  selection 
Radar  terrain  avoidance  mode 


Air-to-Air  Mode 
Gun 

Sparrow  (AIM-7F) 
Sidewinder  (AIM-9L) 
Radar  control 
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V  SUMMARY 


A.  Recorded  Data  Advantages 

The  use  of  a  digital  data  acquisition  system  significantly  improves 
the  quality  of  recorded  measurements.  The  primary  advantages  of  this 
technology  include:  (1)  higher  accuracy  of  recording,  (2)  increased 
number  of  channels  available,  and  (3)  greater  dynamic  range.  This  per¬ 
mits  more  sophisticated  analysis  of  trace  waveforms  than  would  be  possible 
with  analog  data  acquisition  systems.  Subtle  changes  in  waveform  morphology 
can  be  detected  and  analyzed.  In  addition  for  physiological  measurements, 
pattern  recognition  and  artifact  rejection  can  easily  be  accomplished 
during  post-flight  reduction  of  the  recorded  data. 

B.  System  Performance  Assessment 

Today's  technology  can  for  the  first  time  provide  an  operational 
assessment  of  the  complete  airborne  weapon  system,  including  the  aircraft, 
pilot,  and  armament.  It  is  possible  to  describe  aircraft  performance, 
pilot  performance,  and  weapon  performance  quantitatively  and  to  relate 
these  figures  to  mission  success. 

Figure  20  illustrates  the  various  links  that  associate  mission 
success  with  the  three  major  system  components  (aircraft,  pilot,  and 
armament).  For  example,  if  a  mission  were  not  successful  but  the  aircraft 
performed  satisfactorily,  the  problem  may  be  with  the  pilot.  Further 
investigation  may  show  that  blood  flow  to  the  brain  was  reduced  during 
a  high-g  maneuver,  thus  limiting  the  pilot's  ability  to  fly  the  mission 
properly.  Or  it  may  simply  be  that  a  given  aircraft  has  superior  perfor¬ 
mance  (i.e.,  speed  or  turn-rate  advantage)  over  a  given  adversary.  The 
system  described  in  this  paper  is  a  relatively  inexpensive  tool  capable 
of  statistically  identifying  the  individual  parameters  that  limit  system 
performance,  whether  they  are  associated  with  the  aircraft,  the  pilot, 
or  the  armament  system.  Summary  statistics  could  be  generated  to  identify 
consistent  problem  areas  associated  with  poor  performance. 
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AIRCRAFT  PERFORMANCE 


FIGURE  20  SYSTEM  PERFORMANCE  ASSESSMENT 


Specifically,  the  instrumentation  systems  described  could  signifi 
cantly  add  to  the  understanding  of  the  following  questions: 

o  How  can  real  combat  system  performance  be  assessed  on  an  opera 
tional  training  range  in  terms  of  kill  probability, 
survivability,  and  kill  ratio? 

o  What  is  the  weak  link  in  the  airborne  weapon  system — the 
aircraft,  the  pilot,  or  the  armament? 

o  If  pilot  workload  is  the  limiting  factor,  which  component  is 
most  influenced — the  mental  workload  or  the  physical  workload? 

The  TACTS/ACMI  and  the  APMS  instrumentation  systems  together  can 
answer  these  questions. 
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TITLE  This  paper  presents  an  interim  report  on  a  study  designed  to  assess  the 
overall  operational  effectiveness  of  a  night  attack  weapon  system  when 
primary  areas  of  concern  include  aircrew  workload  and  stress. 

PURPOSE  Night  attack  conducted  at  low-altitude  is  a  primary  thrust  of  many  developing 
weapon  systems  (the  B-l ,  F- 16,  MSIP,  F-16E,  F-15E,  the  H-X,  to  name  a  few). 

In  many  of  these  systems  concern  exists  as  to  whether  aircrew  tasking  is 
being  pushed  into  an  area  when  human  restrictions  will  impose  serious  limits 
upon  operational  effectiveness.  Division  of  attention  (mental  and  physical), 
reaction  time,  and  stress  are  all  becoming  more  critical  in  this  mission 
area.  There  is  a  serious  DOD-wide  need  to  eliminate  the  risks  and  uncertainty 
in  these  programs  that  arise  from  these  human  factors  concerns.  The  answer 
has  high  value  if  found  early  in  these  programs  and  high  cost  if  delayed. 

APPROACH  The  study  approach  consisted  of  three  broad  steps.  The  first  was  to  lay  out 
the  critical  issues  that  arise  in  this  mission  area  for  any  manned  system. 

Our  underlying  concept  of  the  aircrew  is  that  of  an  individual  with  flexible 
but  limited  capacity  for  sensory-motor  skills,  mental  calculations/decision¬ 
making,  and  ability  to  function  under  stress.  These  human  capacities, 
different  for  each  individual,  interact  with  specific  tasks  generated  by  the 
characteristics  of  the  weapon  system  and  mission  requirements.  Any  test  for 
operational  effectiveness  must  address  these  three  factors: 

INDIVIDUAL  VARIATIONS 
WEAPON  SYSTEM  CHARACTERISTICS 
MISSION  TASKING/ENVIRONMENT 

Our  second  phase  was  the  establishment  of  an  overall  methodology  and  data 
requirements  based  upon  these  three  functions. 

The  third  step  was  a  survey  of  available  methods/facilities/equipment  for 
the  execution  of  such  a  test. 

CRITICAL  DOD  Directive  5000.3  defines  Operational  Test  and  Evaluation  as: 

"That  test  and  evaluation  conducted  to  estimate  a  system  operational 
effectiveness  and  suitability,  as  well  as  the  need  for  any  modifica- 
tion.  It  is  accomplished  by  operational  and  support  personnel  of 
the  type  and  qualifications  expected  to  use  the  system  when  deployed 
and  is  conducted  in  as  realistic  an  operational  environment  as  possible." 
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Operational  Effectiveness  is  defined  by  AFR  80-4  as: 

"How  well  the  system  performs  its  intended  mission  in  its  intended 
environment." 

These  meanings  translate  into  total  man/machine  system  capability  to  penetrate 
enemy  defenses  and  countermeasures  in  such  a  manner  that  an  appropriate 
number  of  targets  can  be  killed  at  acceptable  levels  of  sortie  effort  and 
attrition  losses.  Therefore,  critical  components  of  the  test  include: 

REALISTIC  TERRAIN/TARGETS 
DAY/NIGHT  OPERATIONS 
ADVERSE  WEATHER 
INTENSE  DEFENSIVE  ARRAY 
ENVISIONED  COUNTERMEASURES 


appropriate  for  the  operational  life  of  the  system. 

STRETCHING  The  cumulative  burden  placed  on  the  human  operator  can  be  envisioned  as  the 
THE  MAN  IN  three  dimensional  integral  volume  under  a  surface  determined  by  the  level  of 
NIGHT  performance  in  each  task  over  the  number  of  tasks  or  "workload"  and  the 

ATTACH  physical  and  psychological  stress  effective  during  the  tasks.  For  each 

mission  there  is  a  "demand  surface,"  and  for  each  individual  aircrew/mission 
combination,  there  is  a  "capability  surface."  The  question,  therefore,  to 
be  answered  is  whether  capability  is  above  or  below  mission  demands  and 
what  state  the  human  operator  is  in  while  performing  the  mission. 

The  night  attack  mission  pushes  the  aircrew  beyond  previous  operational 
tasking  in  several  respects: 

1.  It  requires  precise  split-second  nap  of  the  earth  flying. 

2.  It  requires  the  monitoring  of  multiple  sensors  (radar,  RHAW  gear, 
FUR,  etc.). 

3.  It  removes  normal  cues  previously  utilized  for  low-altitude  flying. 

4.  It  projects  the  system  deep  into  intense  defenses  and  new  counter¬ 
measures. 


SAMPLING  The  operational  effectiveness  test  must  subject  a  representative  cross- 
STRATEGY  section  of  intended  human  operators  to  realistic  mission  tasking  and 
environment.  Various  experimental  controls  must  be  employed  to  insure 
that  the  test  characterizes  the  projected  operational  effectiveness  of  a 
"typical"  unit  in  the  mission  tasking.  Experience,  training,  and  response 
to  stress  must  be  used  in  selecting  test  aircrews.  Sufficient  subjects 
must  be  included  to  represent  individual  variations. 

SYSTEM  The  interactions  of  the  man/machine/mission/environment  are  complex.  Yet, 

PERFORMANCE  they  must  be  faced  in  their  synergistic  combination  since  that  interaction 
ASSESSMENT  is  the  central  issue  of  concern. 
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In  addition  to  variations  in  the  human  operator's  capability,  there  are  natural 
operating  factors  which  interact  to  drive  operational  effectiveness.  Key 
among  these  are  the  trade  offs  between  terrain  clearance  and  speed  utilized 
to  minimize  attrition.  The  low-altitude  contest  between  penetrator  and 
defenses  is  one  of  the  major  evaluations  of  modern  tactical  warfare. 


High-speed,  very  low-altitude  flight  is  possible  over  smooth  terrain,  but 
defenses  can  also  operate  very  low  under  these  circumstances.  Encountering 
rough  terrain  will  force  either  a  climb  or  reduction  in  speed.  All  these 
variations  are  impacted  by  visibility  conditions  and  the  availability  of 
peripheral  cues  (ambient  light). 

For  system's  effectiveness,  these  factors  must  all  be  considered  while 
still  summarizing  to  an  overall  warfighting  capability  and  combat  loss  rate. 

EASURES  In  judging  overall  weapon  system  effectiveness  when  the  primary  concern  is 

F  human  limitation,  we  not  only  assess  performance  in  the  ability  to  accurately 

FFECTIVENESS  find  and  kill  appropriate  numbers  and  types  of  targets,  or  survivability 
rates  from  various  causes,  but  we  must  also  be  aware  of  the  internal  state 
of  the  human  operator  (for  both  reliability  and  combat  sustainability 
reasons). 

Three  strategies  are  available. 

I  -  The. first  is  to  assess  performance  by  observing  differences  of  operators. 

However,  we  have  no  clue  how  close  to  human  limitations  we  are  operating 
or  what  type  of  deterioration  might  occur  if  these  limits  are  exceeded. 

II  -  The  second  strategy  is  to  observe  performance  while  measuring  the 

reserve  capacity  in  cognitive,  sensory-motor,  decision  tasks,  as  well 
as  physical  and  psychological  stress.  This  strategy  leads'to  some 
expectation  in  combat  sustainability. 

Ill-  The  third  strategy  is  to  have  some  criteria  in  reserve  capacity  to 
insure  sustainability  and  then  see  what  combat  performance  is  still 
attainable. 

Both  II  and  III  are  desirable  strategies  for  OT&E. 


HUMAN  In  light  of  the  previous  discussion,  the  data  requirements  are  driven  by 

FACTORS  the  need  to  characterize  operational  measures  of  effectiveness  as  functions 
DATA  of  the  component  capabilities  of  the  individual.  Knowing  these  relation¬ 
ships  for  various  test  conditions  along  a  scale  of  workload/stress 
combinations  provides  a  true  insight  into  the  state  of  the  individual 
as  he  meets  these  varied  demands.  These  trends  also  make  possible  the 
last  critical  step,  that  is  the  characterization  of  potential  combat 
effectiveness  based  upon  test  conditions  that  can  never  reach  the  full 
severity  of  the  next  war.  In  other  words,  we  need  to  extrapolate  to 
WW III  as  best  we  can. 
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If  we  are  to  accurately  characterize  the  man/machine  system,  every  key 
system  function  must  be  duplicated  in  the  test  precisely  as  it  interacts 
with  the  man  and  the  mission  environment.  Further,  and  of  equal  importance, 
the  test  must  provide  realistic  mission  tasking  in  terms  of  terrain, 
weather,  ambient  light,  distance  to  target,  etc.,  as  well  as  the  critical 
interaction  with  enemy  forces.  Finally,  the  proper  instrumentation,  test 
techniques  and  recording  equipment  must  be  set  up  to  capture  both  performance 
and  human  state. 

Test  events  need  to  be  controlled  and  repeated  enough  times  such  that  the 
functional  relationship  between  performance  and  human  capabilities  can  be 
confidently  reported.  Only  when  adequate  data  is  gathered  regarding  each  subject 
and  the  potential  user  population  is  adequately  represented  by  enough  subjects, 
can  we  report  performance  expected  from  personnel  "of  the  type  and  qualifica¬ 
tions  expected  to  use  the  system  when  deployed." 

Having  thus  established  the  requirements  and  characteristics  of  the  desired 
test  we  now  turn  to  an  evaluation  of  different  types  of  test  execution. 

Flight  test  includes  airborne  simulation,  specifically  designed  test 
vehicles,  prototypes,  and  pre-production  hardware. 

Flight  simulator  is  a  ground  based  simulation  centered  around  the  crew 
position  with  appropriate  displays,  controls,  video,  computer  support  and 
scenario/threat  data  bases. 

Task  Analysis  (TA)  is  basically  an  action  path  chart  on  paper  of  the 
tasks/decisions  required  to  perform  the  mission. 

Computer  is  essentially  the  interaction  of  a  "computer  man"  with  a 
TA  input. 

Questionnaires  include  various  subjective  methods  which  require  the 
human  operators  to  "sel f-eval uate"  his  experiences  and  mental  state. 

The  essential  issue  then  for  operational  effectiveness  testing  is  the 
ability  of  these  five  classes  of  tests  to  provide  quantitative  data  for 
the  operational  performance  MOE's  either  directly,  by  inferrence,  or  by 
providing  supporting  information.  Additional  data  on  the  simultaneous 
state  of  the  operator  is  also  desirable.  On  the  chart  an  "X"  indicates 
inability  to  credibly  address  the  MOE  with  that  method.  Other  comments 
within  the  chart  are  self-explanatory. 

Our  evaluation  concluded  that  only  flight  test  and  flight  simulators  can 
credibly  provide  the  performance  measures  needed  to  make  an  overall  system 
evaluation.  This  conclusion  is  reinforced  when  considering  the  major  issue 
to  be  the  cumulative  interaction  of  mission  tasking  on  the  man/machine  system. 

Many  synergetic  effects  are  likely  in  this  complex  scenario  which  can  only  be 
addressed  by  subjecting  potential  operators  to  the  multiple  tasks  required. 

When  the  issues  at  hand  include  division  of  attention,  situation  awareness, 
and  attention  allocation  strategy  under  a  complex  mission  tasking/stress,  it 
is  recommended  to  test  to  that  total  load  and  stress  as  closely  as  practical. 
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The  fundamental  issue  remains  that  short  of  full  automation,  each  of  these 
MOE's  is  determined  by  the  actions/decisions  of  the  operator. 

Essential  elements  of  the  test  include: 

Visual  crosscheck  outside  and  inside  the  cockpit. 

Auditing  perception/feedback. 

Overall  situation  awareness/staying  ahead  of  events. 

Mental  calculating  and  decision  time. 

Piloting  skill  (quickness  +  firmness  of  control). 

Manual  dexterity/switchology. 

Cockpit  control/display  integration. 

Aircrew  psychological  makeup. 

It  is  obvious  from  this  tasking  that  training  and  experience  of  the  aircrew 
are  key  factors  in  this  evaluation. 

Therefore,  two  other  issues  must  be  integrated  into  the  test  design.  First, 
a  representative  cross-section  of  potential  operators  must  be  included  in 
the  evaluation.  Second,  a  history  of  performance  versus  experience  within 
the  test  must  be  kept  as  an  indication  of  training  requirements  when  the 
system  is  deployed. 

Thus,  the  quantifying  of  operational  effectiveness,  and  the  interaction  of 
training  together  lead  to  in-flight  testing  and  ground  based  simulation 
as  the  two  primary  methods  to  be  used. 

COMPARISON  A  comparison  of  flight  test  versus  ground  based  simulation  is  a  study  in 
OF  TEST  technological  and  cost  limitations  rather  than  inherent  restraints.  In 

METHODS  applying  the  data  requirements  against  these  two  methods,  we  find  the 
following  capabilities, 

,  %  T  >  , 

WEAPON  SYSTEM  FIDELITY.  The  ability  to  reproduce  avionics  signal  processing 
is  .really  .pnTy  dependent  upon  generating  those  signals  or  a  reasonable 
facsimile  from  the  simulation.  Thus,  a  terrain  following  radar  cue  to  the 
pilot  can  be  the  result  of  actual  radar  returns  in  flight  or  the  terrain 
data  base  of  a  digital  simulation  as  unmasked  from  present  position 
coordinates.  The  cost  may  be  lower  for  a  ground  based  simulator  since 
only  the  logic  and  display  are  necessary  and  electronic  equipment  does 
not  require  in-flight  rating.  One  key  area  of  technological  concern 
for  the  night  mission  is  the  ability  to  generate  a  realistic  infrared 
video  display  that  truly  interacts  with  cloud/humidity/aerosol  content  as 
selected  from  the  simulator  sonsole. 

SCENARIO  FIDELITY.  The  ability  to  duplicate  World  War  III  or  a  NATO 
conventional  conflict  does  not  currently  exist  for  either  method.  For 
in-flight  testing  true  nap  of  the  earth  flying  is  limited  by  safety 
constraints.  Defense  simulations  or  captured  equipment  is  available  for 
flight  test  but  not  in  sufficient  quantity/qual ity  to  truly  simulate  a 
Warsaw  Pact  defensive  array  either  at  the  FEBA  or  at  a  high  value 
interdiction/strike  target. 
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For  a  ground  based  simulation,  the  terrain  can  be  generated  either  by 
scale  model  board  or  computer  generated  imagery  to  duplicate  actual 
European  geography.  Model  boards  are  severely  limited  in  flight  duration, 
in  that  flight  at  fighter/bomber  speeds  quickly  consumes  the  available 
distance  in  a  matter  of  a  few  minutes.  Computer  generated  imagery  (CGI) 
has  progressed  from  a  very  rough  approximation  of  terrain  with  few 
enhancing  aspects  (trees,  roads,  etc.)  to  textured  imagery  with  a  sizeable 
menu  of  enhancing  objects  on  call  CGI  has  no  real  distance  limitation 
other  than  the  availability  of  topographic  data.  Defensive  arrays  are 
also  limited  only  by  computer  memory  and  processing  power.  As  defensive 
sites  (SAM  or  AAA)  are  unmasked,  their  characteristics  can  be  used  to 
activate  cockpit  radar  homing  and  warning  (RHAW)  gear  indications,  and 
interactive  engagements  can  be  played  to  assess  the  probability  of  kill. 

Finally,  areas  basically  unsafe  for  flight  test  can  be  investigated  in 
simulators.  These  include  serious  overload,  fatigue,  and  potentially  life 
threatening  ground  clearances. 

Flight  does  produce  unique  physical  sensations  that  cannot  be  reproduced 
on  the  ground.  Attempts  to  generate  motion  cues  in  simulators  physically 
with  hydraulic  systems  have  generally  been  very  costly  and  of  limited  use 
due  to  limited  duration  and  negative  cues.  Thus,  other  motion  cues  due  to 
vision,  inflation  of  "g"  suits,  etc.,  have  come  into  use. 

Finally,  the  question  of  whether  the  subject  becomes  wholely  convinced 
and  excited  during  testing  such  that  combat  levels  of  emotional  stress 
are  experienced,  has  caused  doubt  in  the  results  of  operational  testing. 
Emotional  stress  can  induce  heightened  or  suppressed  performance.  Further, 
these  effects  occur  under  different  conditions  for  different  individuals, 
or  even  the  same  individuals  at  different  times.  Various  stress  indicators 
such  as  pulse,  blood  pressure,  respiration  rate,  voice,  etc.,  are  known. 

It  is  reasonable,  therefore,  to  gather  readings  on  these  indications  both  in 
test  and  in  combat.  Over  an  extended  period,  a  functional  pattern  could  be 
established  between  these  indicators  and  performance.  It  is  felt  that 
performance  enhancements  or  degradations  at  various  levels  of  stress  as  a 
pattern  are  far  more  important  data  than  any  single  reading  obtained  by  trying 
to  artifically  duplicate  fear  or  anxiety. 

MULTIPLE  By  measuring  performance  MOE's  under  various  levels  of  stress  indicators 

POINT  for  each  human  operator,  multivariate  analysis  can  be  used  to  predict 

PROTECTION  combat  performance.  Due  to  the  complexity  of  the  interactions  and  the 
influence  of  individual  differences,  a  significant  test  matrix  of 
individuals,  test  conditions,  and  controlled  replications  must  be 
executed.  Only  then  can  the  emerging  factors  influencing  operational 
effectiveness  be  defined.  Multiple  analytical  techniques  can  be  used 
against  these  data  including: 

Simple  trend  analysis 

Constant  2nd/3rd  derivative  assumptions  and  stepwise  integration 
Quadratic  function  definition  by  multiple  points 
Curve  fitting 

Correlation/Regression  Analysis 
Multivariate  Analysis 


ADVANTAGES 

OF 

COMBINED 

METHODS 


These  techniques  call  for  the  use  of  both  ground  based  simulation  and 
flight  test. 

It  is  therefore  the  conclusion  of  this  study  that  both  ground  based 
simulation  and  flight  test  are  required  for  the  operational  effectiveness 
evaluation  of  a  complete  weapon  system. 


Currently,  simulators  have  significant  advantages  in  safety,  cost, 
experimental  control,  and  scenario  intensity.  Flight  test  has  the 
advantage  in  ultimate  realism,  motion,  and  weapons  employment. 

By  using  both  in  an  organized,  sequential  program  the  strongest  features 
of  each  method  can  be  applied.  These  mutually  complimentary  data  gathering 
strengths  not  only  complete  the  loopholes  in  a  single  method,  but  generate 
an  interlocking  pattern  which  can  be  used  to  project  beyond  test  conditions 
toward  combat  expectations. 

Past  studies  have  shown  a  sortie  cost  comparison  of  simulator  versus  flight 
of  one  to  six.  Thus,  if  the  data  from  each  were  equally  valued,  a  typical 
test  would  tend  to  have  more  simulator  sorties  by  the  inverse  ratio.  Time 
and  cost  pressures  could  push  the  ratio  higher. 

In  any  program,  the  use  of  both  methods  would  be  more  cost  effective  than 
reliance  on  either  method  by  itself,  by  leading  to  more  complete  data,  and 
enhanced  use  of  resources. 
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STRETCHING  THE  MAN  IN  NIGHT  ATTACK 
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SAMPLING  STRATEGY 
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WEATHER 


MEASURES  OF  EFFECTIVENESS 
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HUMAN  FACTORS  DATA 
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EVALUATION  OF  POTENTIAL  METHODS 
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ERRORS  INFERRED  OR  DIRECT  INFERRED  INFERRED  COULD  PROVIDE 

RECORDED  RECORDING  INPUT 


COMPARISOM  OF  TEST  METHODS 
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ANXIETY  ACTUAL  FLT  COMBAT  PEACE 

AXIS  COMBAT  COMBAT  SIM 

SIM 

BLOOD,  URINE,  VOICE.  ETC.,  TELL  US  HOW  FAR  ALONG  ANXIETY  AXIS  WE  ARE 


ADVANTAGES  OF  COMBINED  METHODS 
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THEORETICAL  BASIS  FOR  WORKLOAD  ASSESSMENT  RESEARCH  AT  NASA-AMES  RESEARCH  CENTER 


Sandra  G.  Hart 
NASA-Ames  Research  Center 
Moffett  Field,  CA 


SUMMARY 

Workload  may  be  thought  of  as  a  collection  of  experiences,  requirements,  feelings, 
demands,  and  circumstances  that  are  referred  to  in  summary  form  by  the  term 
"workload".  When  one  person  says  that  he  really  worked  hard,  he  may  mean  that 
he  is  physically  tired,  while  another  person  may  provide  a  rating  of  equivalent 
magnitude  because  he  was  required  to  do  more  than  expected,  even  though  his 
actual  output  and  effort  did  not  increase.  There  are  many  factors  associated 
with  the  term  workload  as  it  is  usually  applied  that  each  exist  independently 
and  can  be  analyzed  as  such  most  profitably.  Task  demands  are  just  that  -  -  task 
demands.  No  additional  meaning  or  value  can  be  associated  with  renaming  this 
factor  "workload".  Physical  effort1 and  emotional  stress  are  also  independent, 
unique  entities  that  can  each  be  measured  by  specific  and  unique  assessment 
techniques,  but  again  neither  is  synonomous  with  "workload"  per  se.  Performance 
is  also  an  independent,  important  entity,  but  again  it  is  not  "workload".  Measures 
of  performance  are  most  relevant  to  determining  how  successful  an  individual  was 
in  meeting  task  demands  but  do  not  reflect  how  hard  he  worked,  what  his  expec¬ 
tations  were,  his  stress  level,  the  time  pressure  felt,  and  so  on. 

The  one  factor  that  does  reflect  the  effect  of  all  of  these  factors  on  each 
individual  is  the  subjective  experience  of  workload.  If  an  individual  feels 
loaded,  he  or  she  is.  This  may  be  the  only  factor  in  the  constellation  of 
elements  variously  called  "workload"  that  is  purely  "workload"  and  nothing 
else.  This  subjective  experience  is  obviously  derived  from  the  other  factors  -  - 
task  demands,  success  in  meeting  demands,  effort,  and  so  on  -  -  but  it  is 
the  product  of  a  weighting  process  that  may  be  unique  to  each  individual.  The 
weights  or  importance  that  each  individual  places  on  the  various  elements  that 
may  affect  his  experience  of  workload  may  differ  from  person  to  person,  although 
they  should  be  fairly  consistent  within  an  individual.  For  example  one  indi¬ 
vidual  may  consider  his  workload  to  be  defined  by  the  demands  of  a  task  whereas 
another  person  may  only  rate  workload  as  high  if  he  is  physically  tired,  and  yet 
another  might  primarily  base  his  estimate  on  his  ability  to  meet  the  demands  of 
a  task,  rather  than  the  amount  of  the  demands  or  the  effort  exerted. 

By  determining  what  factors  enter  into  this  weighting  process  and  how  they  are 
combined,  it  may  be  possible  to  develop  methods  to  assess  this  subjective 
factor  -  -  the  one  element  that  may  be  uniquely  "workload"  -  -  to  use  in  the 
interpretation  of  subjective  ratings,  variation  in  performance,  and  physiological 
recordings.  The  assumption  is  that  if  a  person  feels  loaded  -  -  he  is  -  -  and 
that  this  will  not  only  affect  his  or  her  subjective  evaluations  of  workload  but 
also  physiological  measures  of  stress,  arousal,  fatigue',  etc.  and  the  individual's 
ability  to  perform  the  primary  task  as  well  as  additional  tasks  effectively. 
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''workload* 

WORKLOAD  IS  A  SUBJECTIVE  EXPERIENCE  CAUSED  BY  EXTERNAL  AND 
INTERNAL  FACTORS  SUCH  AS  MOTIVATION,  ABILITY,  EXPECTATIONS, 
TRAINING,  TIMING,  STRESS,  FATIGUE,  AND  CIRCUMSTANCES  IN 
ADDITION  TO  THE  NUMBER,  TYPE,  AND  DIFFICULTY  OF  TASKS  PER¬ 
FORMED,  EFFORT  EXPENDED,  AND  SUCCESS  IN  MEETING  REQUIREMENTS, 


A  SUBJECTIVE  EXPERIENCE  OF  WORKLOAD 


THE  AREA  OF  THE.  ABOVE  FIGURE  REPRESENTS  A  SUBJECTIVE  EXPERIENCE  OF 
WORKLOAD  AS  MEASURED  BY  A  RATING,  OBJECTIVE  MEASURE  OR  PHYSIOLOGICAL 
INDICE, 
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THERE  ARE  SEVERAL  DIFFERENT  DIMENSIONS 
THAT  COMBINE  TO  CREATE  THE 
EXPERIENCE  OF  WORKLOAD 


r 

TASK-RELATED 


PILOT-RELATED 


OUTCOME 

RELATED 


The  area  of  the  circle  represents  the  magnitude  of  workload  experienced 
The  different  areas  and  colors  represent  the  dimensions  of  that  experience 
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COMPONENTS  OF  THE  DIMENSIONS  THAT 
CREATE  THE  EXPERIENCE  OF  WORKLOAD 


Time  pressur 


Amount  of 
task  demand 


Type  of  tasl 
demand 


.  e  /  Task 

complexity 


V 


|  Boredom 

[ 

[  Training 

p 

Mental 

Busyness 

Physical 

Busyness 

\ 

/  Feedback 
I  during  task 


Motiva¬ 

tion 


Well¬ 

being 


Type  of 
effort 


Amount  oq 
effort 


Environment- 


Fatigue 


performance^ 


Emotional] 
stress 


THE  AREA  OF  THE  CIRCLE  REPRESENTS  THE  MAGNITUDE  OF  WORKLOAD  EXPERIENCED 
THE  SHAPE  AND  COLOR  OF  THE  COMPONENTS  REPRESENTS  THEIR  SOURCE  (TASK/ 

operator/outcome) 
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THE  MAGNITUDE  AND  NATURE 
OF  THE  COMPONENTS  MAY  VARY 


THE  AREA  OF  THE  CIRCLE  REPRESENTS  THE  MAGNITUDE  OF  WORKLOAD  EXPERIENCED 

THE  SIZE  OF  EACH  COMPONENT  INDICATES  HOW  MUCH  OF  IT  WAS  PRESENT 
AND  RELEVANT  IN  A  GIVEN  EXPERIENCE 


THE  SHAPE  AND  COLOR  OF  EACH  COMPONENT  REPRESENTS  ITS  SOURCE  (TASK- 
OPERATOR-  OR  OUTCOME-RELATED) 
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THE  COMPONENTS  MAY  OR  MAY  NOT 

(these  interrelationships  may  be  causal  or  coincidental,  positive  or  negative) 


The  shape/color  of  a  component  represents  its  source(task-  operator-  or 
outcome-related) 

The  size  of  the  component  represents  the  "amount"  of  that  component  present 

The  intersection  between  components  represents  covariation 

The  area  of  the  circle  represents  the  magnitude  of  the  workload  experienced 
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THE  IMPORTANCE  GIVEN  TO  EACH  COMPONENT 
IN  ESTIMATING  OR  REACTING  TO  WORKLOAD 


ALSO  VARIES 


The  shape/color  of  components  represents  source  (task/operator/outcome) 

The  size  represents  the  "amount"  of  that  component  present 

The  intersection  between  components  represents  covariation 

The  height  of  components  represents  the  importance  or  weight  they  are 
given  by  one  individual 


The  area  of  the  circle  represents  the  magnitude  of  the  workload  experienced 
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WORKLOAD  IS  RATED  AS  LOW  WORKLOAD  IS  RATED  AS  HIGH 
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INITIAL  RESEARCH  GOALS: 
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METHOD  FOR  OBTAINING  INFORMATION  ABOUT  THE  IMPORTANCE  OR  WEIGHT 
PLACED  ON  DIFFERENT  DIMENSIONS  IN  AN  INDIVIDUAL'S  CONCEPT  OF  WORKLOAD 
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BI-POLAR  ADJECTIVE  SCALES 
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DETERMINATION  OF  SENSITIVE  MEASURES  OF 
PILOT  WORKLOAD  AS  A  FUNCTION  OF  THE 

TYPE  OF  ILOTING  TASK 


WALTER  W.  WIERWILLE 

Virginia  Polytechnic  Institute  and  State  University 
Blacksburg,  Virginia  24061 


SUMMARY 

The  purpose  of  our  present  work,  sponsored  by  NASA-AMES,  is  to  examine  the 
sensitivity,  intrusion,  and  transferability  of  a  variety  of  workload  assessment 
techniques.  The  study  will  use  four  different  simulated  piloting  tasks,  em¬ 
phasizing  psychomotor,  perceptual,  mediational,  and  communications  aspects .  Pi¬ 
lot  loading  levels  will  be  systematically  adjusted.  Our  simulation  facility  is  a 
GAT-1B  that  has  been  modified  and  instrumented  for  workload  estimation  techniques 
measurement.  The  flight  simulator  .tself  has  three  degrees  of  physical  motion 
and  a  full  complement  of  IFR  instruments. 

Recently  we  completed  the  experiment  emphasizing  the  psychomotor  aspect  of 
flight.  Instrument-rated  pilots  flew  instrument  approaches  under  three  combined 
settings  of  the  independent  variable:  increasing  turbulance  and  decreasing  longi¬ 
tudinal  stability.  Twenty  different  workload  measures  were  taken  between  the  outer 
and  middle  markers,  only  five  of  which  showed  statistically  reliable  changes  as  a 
function  of  the  independent  variable.  Included  in  the  five  were:  two  rating 
scales,  one  measure  of  control  movement  activity,  pulse  rate,  and  one  measure  of 
time  estimation.  The  results  of  the  experiment  are  to  some  extent  surprising,  for 
they  indicate  that  several  "accepted"  measures  of  workload  are  not  reliably  sensi¬ 
tive  to  the  kinds  of  psychomotor  load  which  pilots  encounter. 
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We  are  currently  planning  the  perceptual  and  mediational  (cognitive)  experi¬ 
ments,  and  we  expect  to  have  the  results  of  these  two  experiments  in  mid-1982. 

INTRODUCTION 

The  increasing  complexity  of  aircraft  systems  and  the  changing  roles  of  pi¬ 
lots  and  other  aircrew  personnel  have  resulted  in  the  need  for  techniques  to  mea¬ 
sure  operator  workload  in  a  wide  range  of  situations  and  tasks.  One  need  only 
initiate  a  preliminary  survey  of  the  literature  on  operator  workload  assessment 
techniques  to  discover  that  a  voluminous  mass  of  information  has  accumulated  rap¬ 
idly  in  the  past  two  decades.  However,  major  reviews  of  this  literature  have  con¬ 
cluded  that  while  workload  research  has  advanced  in  both  scope  and  technology, 
basic  questions  remain  to  be  answered  for  the  practitioner  who  wishes  to  select 
workload  measures  for  a  given  application  (Wierwille  and  Williges,  1978),  Hicks 
and  Wierwille  (1979)  have  pointed  out  that,  in  particular,  the  lack  of  information 
on  the  relative  sensitivity,  the  degree  of  intrusion,  and  the  range  of  transfer- 
ability  of  individual  techniques  makes  it  difficult  for  a  practitioner  to  select 
workload  estimation  techniques  for  a  given  task. 

The  purpose  of. our  present  work  is  to  help  fill  the  need  for  practical  in¬ 
formation.  Specifically,  techniques  for  measurement  of  pilot  workload  arebeing 
selected  and  compared  to  determine  their  relative  sensitivity,  intrusion,  and 
trans  f er ab ili ty . 

Before  proceeding  with  further  discussion  and  results  of  our  experiments  up 
to  the  present,  it  would  be  helpful  to  define  the  terms,  sensitivity,  intrusion, 
and  transferability.  Sensitivity  can  be  defined  as  the  relative  ability  of  a 
workload  estimation  technique  to  discriminate  statistically  significant  differ¬ 
ences  in  operator  loading.  High  sensitivity  requires  diseriminable  changes  in 
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the  score  means  as  a  function  of  load  level  and  low  variation  of  the  scores  about 
the  means o  Intrusion  can  be  defined  as  an  undesirable  change  in  the  task  for 
which  workload  is  being  estimated,  resulting  from  the  introduction  of  a  workload 
measurement  technique  or  apparatus.  And,  transferability  is  the  relative  ability 
of  a  workload  estimation  technique  to  remain  sensitive  when  being  applied  in  situ¬ 
ations  requiring  different  operator  behaviors  or  skills „ 

Unfortunately,  there  has  been  no  definitive  major  effort  aimed  at  sensitivity, 
intrusion,  and  transferability.  As  a  result,  progress  in  determining  which  work¬ 
load  estimation  techniques  should  be  selected  for  a  given  application  has  been 
painfully  slow.  When  asked  which  techniques  are  sensitive  in  a  given  piloting 
situation,  an  honest  workload  researcher  has  difficulty  responding,,  The  danger 
is  that  in  a  given  application  insensitive  techniques  may  be  used.  These  tech¬ 
niques  would  show  no  substantial  change  in  workload  when  in  fact  there  is  a  change. 
Unless  one  knows  that  a  technique  is  sensitive  in  a  given  situation,  one  has  no 
assurance  the  evaluation  of  workload  in  an  experimental  situation  will  result  in 
definitive  conclusions „ 

In  our  work  we  have  arbitrarily  divided  piloting  behavior  into  four  cate¬ 
gories:  psychomotor,  perceptual,  mediational,  and  communications „  These  four 
behaviors  are  those  suggested  by  Berliner,  Angell,  and  Shearer  (1964),  in  their 
list  of  universal  operator  behaviors  (See  also  Wierwille  and  Williges,  1980). 
Clearly,  other  task  taxonomy  categories  might  have  been  chosen.  However,  the 
Berliner,  et  al  categories  do  appear  to  reflect  the  major  categories  of  behaviors 
exhibited  by  pilots  and  other  aircrew  members „ 

Our  evaluation  of  workload  estimation  techniques  in  psychomotor  tasks  has 
been  completed.  Results  of  the  study  will  be  summarized  in  the  following  sections 
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of  this  paper.  More  complete  descriptions  appear  on  Connor  1981  and  Connor  and 
Wierwille,  1982. 

Presently,  we  are  in  the  final  stages  of  planning  for  the  experiments  em¬ 
phasizing  perceptual  behavior  and  mediational  behavior.  In  both  cases  pilots 
will  fly  the  simulator  in  the  simple  task  of  maintaining  heading,  airspeed,  and 
altitude.  For  the  perceptual  task  they  will  also  perform  a  forced-pace  visual 
search  task  presented  through  the  windscreen  using  an  Ektagraphic  display.  The 
complexity  of  the  search  task  will  become  the  independent  variable.  For  the 
mediational  task,  navigation  problems  will  be  presented.  The  problems  will  be 
forced-pace,  but  will  not  require  computational  aids.  We  expect  to  have  the  re¬ 
sults  of  these  two  experiments  in  mid-1982a  After  these  experiments  have  been 
completed  we  will  also  plan  and  carry  out  a  simulated  task  involving  communica¬ 
tions.  In  all  the  experiments,  loading  level  will  be  the  independent  variable, 
and  technique  scores  will  be  the  dependent  variables.  By  conducting  the  experi¬ 
ments  using  this  philosophy,  we  can  obtain  direct  comparisons  of  the  sensitivity 
of  various  techniques.  Transferability  will  be  evaluated  by  determining  which 
(if  any)  techniques  remain  sensitive  from  one  experiment  to  another  .  And,  since 
primary  task  measures  are  taken  for  all  techniques,  intrusion  can  be  determined 
by  comparision  of  primary  task  measures  with  and  without  the  other  workload  mea¬ 
surement  techniques. 

Clearly,  the  studies  we  are  performing  must  necessarily  be  limited  in  scope, 
and  they  will  not  answer  all  important  questions  about  sensitivity,  intrusion, 
and  transferability.  Nevertheless  we  believe  they  will  be  very  helpful  to  prac¬ 
titioners  who  must  evaluate  workload  in  realistic  aircraft  environments. 
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REVIEW  OF  THE  EXPERIMENT  ON 
PSYCHOMOTOR  LOAD 


Subjects 

Six  male  instrument-rated  pilots  served  as  subjects  in  this  experiment.  The 
flight  time  of  the  subjects  ranged  from  500  to  2700  hours  with  a  mean  of  1300  hours. 

Apparatus 

The  primary  apparatus  in  this  experiment  was  a  modified  flight  task  simulator 
(Singer  Link,  Inc.,  General  Aviation  Trainer,  GAT-1B) .  The  simulator  had  three  de¬ 
grees  of  freedom  of  motion  (roll,  pitch,  and  yaw).  Translucent  blinders  were  used 
to  cover  the  windows  of  the  simulator  to  reduce  outside  distractions  and  cues  and 
to  aid  in  the  control  of  cockpit  illumination. 

Several  modifications  to  the  flight  simulator  were  made  for  the  experiment. 
These  modifications  permitted  primary  task  load  manipulation,  secondary  task  op¬ 
erations,  response  measurement,  and  scoring.  Primary  task  load  manipulation  was 
accomplished  by  changing  aircraft  pitch  stability  and  random  windgust  disturbance 
level  simultaneously.  Three  load  conditions  were  developed:  low,  medium,  and 
high,  as  shown  in  Table  1. 

Table  2  provides  a  list  of  the  workload  measurement  techniques  selected  for 
inclusion  in  the  present  study.  These  techniques  were  selected  on  one  of  two 
bases.  First,  evidence  was  found  which  indicated  that  the  measures  might  be  sen¬ 
sitive  indicators  of  pilot  workload  in  both  simulated  and  operational  flight. 

Second,  previous  research  had  shown  that  these  measures  could  be  useful  in  a  va¬ 
riety  of  tasks  relevant  to  the  flight  environment.  A  review  of  the  twenty  tech¬ 
niques  selected  can  be  found  in  Connor  (1981). 
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Experimental  Design 

A  complete  3  x  20  wi thin-subject  design  was  used  for  the  sensitivity  analysis. 
Load  was  the  factor  with  three  levels .  Measurement  technique  (Table  2)  was  the 
factor  with  twenty  levels. 

Workload  measures  from  different  techniques  were  taken  simultaneously  on  some 
of  the  data  collection  runs.  Only  those  measures  which  were  not  likely  to  affect 
each  other  were  taken  simultaneously.  Table  3  shows  the  scheme  used  for  combining 
different  measurement  techniques  for  data  collection.  The  combination  of  measure¬ 
ment  techniques  shown  in  the  table  was,  to  an  extent,  based  on  previous  investiga¬ 
tions  of  workload.  Hicks  and  Wierwille’s  (1979)  study  supported  the  combination 
in  condition  2.  The  two  rating  scales  were  administered  in  separate  measurement 
conditions  to  prevent  the  ratings  on  one  scale  from  biasing  the  ratings  on  the 
other  scale.  The  secondary  task  measures  were  divided  among  several  conditions 
because  of  potential  intrusion  and  interference.  Vocal  measures  were  recorded  from 
the  two  secondary  tasks  which  required  a  verbal  response  as  per  Schiflett  and 
Loikith's  (1979)  recommendation. 

It  should  be  noted  that  primary  task  measures  were  recorded  on  all  subjects 
and  on  all  data  collection  flights  for  the  intrusion  analysis.  However,  only  data 
from  measurement  condition  1  were  used  for  the  sensitivity  analysis  of  the  primary 
task  measures. 

The  intrusion  analysis  was  designed  to  examine  the  effect  of  measurement  con¬ 
dition,  and  the  interaction  of  measurement  condition  with  load  on  primary  task 
performance.  Data  for  all  primary  task  measures  were  therefore  collected  for  each 
flight  performed  in  the  six  measurement  conditions. 
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General  Procedure 


After  receiving  instructions,  subjects  flew  nine  familiarization  flights  in 
the  simulator.  These  flights  were  similar,  but  not  the  same  as,  the  data  collection 
flights.  All  subjects  flew  the  familiarization  flights  in  the  same  order .  Steady 
crosswinds  were  introduced  for  each  run,  and  subjects  were  given  heading  corrections. 

After  the  familiarization  session,  the  subjects  participated  in  three  data 
collection  sessions.  The  familiarization  session  and  each  data  collection  session 
were  held  on  a  different  day. 

Each  data  collection  session  consisted  of  two  sets  of  a  warm-up  practice 
flight  and  three  data  collection  flights.  The  practice  flight  was  the  same  as  the 
first  data  collection  flight.  Since  the  data  collection  flights  were  counterbal¬ 
anced,  equal  amounts  of  practice  were  provided  for  the  low,  medium,  and  high  load 
conditions.  The  data  collection  flights  also  contained  steady  crosswind  conditions, 
for  which  the  subject  was  given  heading  correfctions.  The  purpose  of  Introducing 
steady  crosswinds  was  to  disguise  the  load  conditions,  thereby  requiring  subjects 
to  fly  each  flight  as  a  separate  entity. 

Flight  Task  Procedures 

The  flight  task  in  this  experiment  was  an  Instrument  Landing  Systems  (ILS) 
approach  to  the  Seaport  Beach  runway  (29L)  which  is  instrumented  in  the  Singer 
Link  GAT-1B  aircraft  simulator.  Prior  to  the  beginning  of  a  flight,  the  simulated 
aircraft  was  positioned  5  miles  outbound  from  the  Seaport  Beach  outer  marker  on  the 
108  degree  radial,  heading  into  the  wind.  When  ready  to  begin,  the  experimenter 
informed  the  subject  of  the  wind  direction  and  speed,  and  gave  him  a  heading  cor- 
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rection  for  the  crosswind.  When  contacted  by  the  experimenter,  the  subject  took 
off  and  climbed  to  2000  feet.  The  subject  then  flew  directly  to  the  outer  marker 
by  following  the  localizer  at  100  miles  per  hour  until  the  glide  slope  was  inter¬ 
cepted.  Upon  interception  of  the  glide  slope,  the  subject  reduced  airspeed  to  80 
miles  per  hour  and  proceeded  down  the  glide  slope  while  following  the  localizer  to 
a  landing.  Data  were  recorded  between  the  outer  and  middle  markers.  For  the  opin¬ 
ion  measures,  subjects  gave  ratings  for  the  flight  segment  between  the  outer  and 
middle  markers  immediately  after  landing  and  parking  the  simulated  aircraft. 

Results 


Sensitivity  Analysis 

The  computed  scores  for  each  technique  were  first  converted  to  Z-scores  (nor¬ 
malized  scores)  so  that  technique  measure  units  would  not  affect  the  sensitivity 
analysis.  Subsequently,  an  overall  analysis  of  variance  was  performed  on  the 
scores.  Since  Z-scores  were  used,  a  technique  main  effect  was  not  possible.  A 
significant  main  effect  of  load  was  found,  F  (2,10)  = 5.34,  £  <  0.0001,  and  a  sig¬ 
nificant  load  by  technique  interaction  was  found,  I?  (38,190)  =  2.76,  p  _<  0.05. 

The  load  by  technique  interaction  indicated  that  the  measurement  techniques 
were  differentially  sensitive  to  load.  Therefore,  individual  ANOVAs  were  used 
to  isolate  the  sensitive  techniques. 

The  individual  ANOVAs  indicated  that  five  of  the  twenty  measures  were  sensi¬ 
tive.  They  were  the  Cooper-Harper  scale  £  (2,10)  =  16.39,  £  =0,0007;  the  Work¬ 
load  Compensation-Interference/Technical  Effectiveness  (WCI/TE)  scale,  _F  (2,  10) 

=  31.15,  £  <0.0001;  the  time  estimation  standard  deviation,  F(2,10)  =  5.69, 
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jd  =0.022;  the  pulse  rate  mean,  F  (2,10)  =  8.89,  £  =  0.006;  and  the  control  movements 
measure,  F  (2,10)  =  33.84,  £  <  0.0001.  The  normalized  means  for  each  technique  are 
plotted  in  Figures  1  through  5  as  a  function  of  load. 

Newman-Keuls  comparisons  were  then  performed  on  the  normalized  means  of  the 
sensitive  measures.  The  comparisons  included  low  vs.  medium,  medium  vs.  high,  and 
low  vs.  high  load  conditions.  Results  indicated  that  all  differences  were  signifi¬ 
cant  at  £  <  0.05,  except  for  pulse-rate  mean  (low  vs.  medium  and  medium  vs.  high) 
and  time  estimation  standard  deviation  (low  vs.  high). 

A  logical  classification  of  techniques  based  on  demonstrated  sensitivity  was 
generated  from  an  examination  of  the  Newman-Keuls  comparisons,  as  shown  in  Table 
4.  Techniques  which  demonstrated  sensitivity  to  all  pairs  of  load  conditions  (i.e., 
low  vs.  medium,  medium  vs.  high,  and  low  vs.  high)  were  included  in  class  I.  These 
measures  are  preferred  over  other  techniques  which  demonstrated  only  partial  sensi¬ 
tivity,  or  no  sensitivity  in  the  present  study.  Techniques  which  showed  sensitivity 
to  some  differences  in  load  conditions  (but  not  all)  were  included  in  class  II. 

These  measures  are  less  preferred  than  class  I  techniques,  but  are  more  preferred 
than  class  III  techniques.  Class  III  techniques  did  not  demonstrate  sensitivity 
to  load  in  the  present  study.  This  class  includes  all  techniques  except  those  in 
class  I  and  class  II. 

One  possible  reason  that  only  five  of  the  twenty  workload  assessment  techniques 
demonstrated  sensitivity  in  the  present  study  is  that  the  other  techniques  simply 
required  a  greater  number  of  subjects  to  show  a  significant  effect  of  load.  It  is 
possible  to  estimate  the  sample  size  required  to  detect  a  reliable  load  effect  for 
a  given  workload  assessment  technique  at  specified  levels  of  significance  and  power. 
These  calculations  were  performed  for  those  techniques  which  did  not  demonstrate 
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sensitivity  in  the  present  study,  to  provide  an  indication  of  the  practical  costs 
of  achieving  statistical  significance 0  The  procedure  used  for  estimating  the  sam¬ 
ple  size  required  for  finding  sensitivity  is  described  by  Bowker  and  Lieberman 
(1959).  Sample  sizes  were  estimated  for  a  significance  level  of  0.05  and  for  a 
power  of  approximately  0.80.  The  results  of  these  estimates  are  presented  in 
Table  5. 

Intrusion  Analysis 

The  equipment  and  procedures  used  for  some  workload  assessment  techniques  may 
interfere  with  performance  on  the  primary  (flight)  task.  In  the  present  experi¬ 
ment,  data  for  the  twenty  measurement  techniques  were  recorded  in  six  measurement 
conditions  as  shown  in  Table  3.  These  six  measurement  conditions  differed  in  the 
equipment  and  procedures  used  for  data  collection.  The  purpose  of  the  intrusion 
analysis  was  to  examine  the  effect  of  these  measurement  conditions  on  primary  task 
performance. 

The  equipment  and  procedures  used  in  measurement  condition  1  were  assumed  to 
be  unobtrusive  to  primary  task  performance.  Primary  task  performance  in  this  con¬ 
dition  was  therefore  used  as  a  standard  of  comparison  for  primary  task  performance 
on  the  other  five  measurement  conditions.  The  measures  of  primary  task  performance 
which  were  used  for  these  comparisons  included  scores  on  localizer  rms  error, 
glide  slope  rms  error,  and  control  movements  per  second. 

A  multivariate  analysis  of  variance  (MANOVA)  was  performed  to  examine  the  ef¬ 
fect  of  condition,  load,  and  the  interaction  of  condition  and  load  on  the  primary 
task  measures.  Only  the  main  effect  of  load  was  found  to  be  significant  F_  (2,10) 

=  9.42,  £  =  0.0002.  Because  there  was  no  significant  effect  of  condition  nor  sig- 
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nificant  interaction  of  condition  with  load,  it  can  be  concluded  that  the  physio¬ 
logical  measuring  equipment  and  the  secondary  tasks  did  not  significantly  affect 
pilot  performance  in  terms  of  the  three  primary  task  variables. 

Conclusions 

This  study  has  shown  that  five  measures  of  workload  estimation  were  sensitive 
indicators  of  load  in  a  piloting  task  that  is  predominantly  psychomotor  in  nature. 
Another  fifteen  measures,  believed  to  be  "good"  measures  of  workload,  showed  no 
reliable  effect.  The  main  conclusion  that  must  be  drawn  from  the  study  is  that 
few  measures  are  sensitive  to  psychomotor  load. 

Of  the  five  techniques  demonstrating  sensitivity,  only  three  exhibited  mono¬ 
tonic  score  increases  with  load  as  well  as  statistically  reliable  differences  be¬ 
tween  all  pairs  of  load  levels.  Consequently,  only  the  three  meet  all  criteria 
for  sensitivity  to  psychomotor  load.  These  class  I  techniques  are  the  ones  that 
are  recommended  for  measurement  of  psychomotor  load: 

Cooper/Harper  ratings, 

WCI/TE  ratings,  and 
Control  movements  per  second. 

The  other  two  techniques  showed  sensitivity  to  psychomotor  load,  but  did  not  dis¬ 
criminate  between  all  pairs  of  load  levels.  These  class  II  techniques  are: 

Time  estimation  standard  deviation,  and 
Pulse  rate  mean. 

These  measures  would  be  helpful  in  evaluating  psychomotor  load,  but  they  should 
not  be  relied  on  exclusively.  At  least  one  class  I  technique  should  also  be  used 
in  conjunction  with  these  measures. 
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It  is  worth  noting  that  only  two  opinion  measures  were  taken  in  the  present 
experiment,  and  both  proved  sensitive .  This  suggests  that  well-designed  rating 
scales  are  among  the  best  of  techniques  for  evaluating  psychomotor  loado  In  re¬ 
gard  to  the  primary  task  measures,  the  control  movements  measure  alone  was  sensi¬ 
tive.  However,  this  measure  is  also  the  only  primary  task  measure  which  reflected 
"strategy"  of  the  pilot.  Consequently,  one  could  speculate  that  selecting  a  pri¬ 
mary  task  measure  that  reflects  strategy  will  most  likely  result  in  good  sensitivity. 

Fifteen  (techniques)  measures  showed  no  reliable  change  as  a  function  of  load. 
When  these  fifteen  measures  were  subjected  to  a  power  analysis  to  determine  sample 
size,  the  number  of  subjects  required  ranged  from  12  to  well  over  100  (Table  5) 0 
One  can  only  conclude  that  at  best  the  fifteen  measures,  as  taken,  are  much  less 
sensitive  to  psychomotor  load  than  the  five  appearing  in  Classes  I  and  II.  Of 
course,  there  is  always  the  possibility  that  the  measures  would  be  sensitive  to 
loading  along  other  dimensions  of  human  performance,  such  as  psychomotor  tasks  of 
a  different  nature,  or  mediational  or  cognitive  tasks,  for  example. 

In  regard  to  intrusion,  this  experiment  showed  that  no  significant  interfer¬ 
ence  occurred  for  the  physiological  measures  or  for  the  secondary  task  measures. 
Performance  as  measured  using  three  primary  (flight)  task  measures  showed  no  re¬ 
liable  changes  as  a  function  of  addition  of  these  measures. 

In  general,  the  results  of  the  experiment  show  that  there  are  wide  variations 
in  the  sensitivity  of  workload  estimation  measures.  Great  care  must  be  taken  in 
selecting  measures  for  a  given  experiment.  Otherwise,  it  is  possible  that  no 
changes  in  workload  will  be  found,  when  indeed  there  are  changes. 


482 


References 


Berliner,  C.,  Angell,  Dc,  and  Shearer,  D,  J.  Behaviors,  measures,  and  instruments 
for  performance  evaluation  in  simulated  environments.  Paper  presented  at  the 
Symposium  and  Workshop  on  the  Quantification  of  Human  Performance,  Albuquerque, 
New  Mexico,  August,  1964. 

Bowker,  A0  H.  and  Lieberman,  G.  J.  Engineering  statistics.  New  Jersey:  Prentice- 
Hall,  Inc.,  1959. 

Connor,  S.  A.  A  comparison  of  pilot  workload  assessment  techniques  using  a  psycho¬ 
motor  task  in  a  moving  base  aircraft  simulator.  Master’s  Thesis,  Virginia  Poly¬ 
technic  Institute  and  State  University,  Blacksburg,  Virginia,  October,  1981. 

Connor,  S.  A.  and  Wierwille,  W.  W.  Comparative  evaluation  of  twenty  pilot  workload 
assessment  measures  using  a  psychomotor  task  in  a  moving  base  simulator.  Moffett 
Field,  CA:  NASA-Ames  Research  Center,  (Forthcoming  report). 

Hicks,  T.  G„  and  Wierwille,  W„  W.  Comparison  of  five  mental  workload  assessment 
procedures  in  a  moving  base  driving  simulator.  Human  Factors,  1979,  21,  129-143. 

Schiflett,  S.  G„  and  Loikith,  G.  J.  Voice  stress  as  a  measure  of  operator  workload. 
Patuxent  River,  Maryland:  Naval  Air  Test  Center,  Technical  Memorandom  TM  79-3 
SY,  December  31,  1979. 

Wierwille,  W.  W.  and  Williges,  R.  C.  Survey  and  analysis  of  operator  workload 
assessment  techniques.  Blacksburg,  Virginia:  Systemetrics,  Inc.  Report  No. 
S-78-101,  September,  1978. 

Wierwille,  W.  W„  and  Williges,  B.  H„  An  annotated  bibliography  on  operator  mental 
workload  assessment.  Patuxent  River,  Maryland:  Naval  Air  Test  Center  Report 
No.  SY-27R-80,  March,  1980. 


Acknowledgements 

The  author  wishes  to  thank  Mr.  Sidney  A.  Connor,  who  carried  out  the  psycho¬ 
motor  workload  experiment,  and  Mrs.  Sandra  Hart,  NASA-Ames  Research  Center,  for 
helpful  technical  suggestions. 


483 


TABLE  1 


Primary  Task 

Load 

Conditions 

Low 

LOAD  CONDITION 

Medium 

High 

RANDOM  GUST  LEVEL 

Low 

Medium 

High 

Estimated 

Std.  Dev.  (mph) 

0 

2.7 

5.9 

PITCH  STABILITY 

High 

Medium 

Low 

ac  Control  input  to  pitch 
rate  output  equivalent 
gain  (degrees /s  per  % 
of  control  range) 

0.522 

3.560 

7.83 

b.  Control  input  to  pitch 
rate  output  equivalent 
time  constant  (s) 

0.097 

0.660 

1.45 
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TABLE  2 


Workload  Assessment  Techniques  Which  Were  Tested  in  the 
Present  Experiment 


OPINION 

1.  Cooper -Harper  Scale 

2.  WCI/TE  Scale 

SPARE  MENTAL  CAPACITY 

3.  Digit  Shadowing  (%  errors) 

4.  Memory  Scanning  (Mean  time) 

5.  Mental  Arithmetic  (%  errors) 

6.  Time  Estimation  Mean  (Seconds) 

7.  Time  Estimation  Standard  Deviation  (Seconds) 

8.  Time  Estimation  Absolute  Error  (Seconds) 

9.  Time  Estimation  RMS  error  (Seconds) 

PHYSIOLOGICAL 

10.  Pulse  Rate  Mean  (Pulses  per  minute) 

11.  Pulse  Rate  Variability  (Pulses  per  minute) 

12.  Respiration  Rate  (Breath  cycles  per  minute) 

13.  Pupil  Diameter  (Normalized  units) 

14.  Voice  Pattern  (Digit  Shadowing  Task) 

15.  Voice  Pattern  (Mental  Arithmetic  Task) 

EYE  BEHAVIOR 

16.  Eye  Transition  Frequency  (Transitions  per  minute) 

17.  Eye  Blink  Frequency  (Blinks  per  minute) 

PRIMARY  TASK 

18.  Localizer  RMS  Angular  Position  Error  (Degrees) 

19.  Glide  Slope  RMS  Angular  Position  Error  (Degrees) 

20.  Control  Movements  per  second 

(Aileron  +  Elevator  +  Rudder) 
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TABLE  3 


Combination  of  Measurement  Techniques 
for  Data  Collection 


Measurement  Condition 

Measurement  Techniques 

1. 

Cooper-Harper  Scale 

Pupil  Diameter 

Eye  Transition  Frequency 

Eye  Blink  Frequency 
Localizer  RMS  Error 

Glide  Slope  RMS  Error 
Control  Movements 

2. 

WCI/TE  Scale 

Pulse  Rate  Mean 

Pulse  Rate  Variability 
Respiration  Rate 

3. 

Digit  Shadowing 

Voice  Pattern 

4. 

Memory  Scanning 

5. 

Mental  Arithmetic 

Voice  Pattern 

6. 

Time  Estimation 
(Mean) 

(Std.  Dev.) 

(Abs.  Error) 

(RMS  Error) 
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TABLE  4 


Logical  Classification  of  Techniques 
Based  on  Demonstrated  Sensitivity 


Class  I:  Complete  Sensitivity  Demonstrated 
Cooper-Harper  Scale 
WCI/TE  Scale 

Control  Movements/Unit  Time 

Class  II:  Some  Sensitivity  Demonstrated 
Time  Estimation  Standard  Deviation* 

Pulse  Rate  Mean  ** 

Class  III:  Sensitivity  Not  Demonstrated 
All  Other  Techniques  (See  Table  5  ) 


*Double  valued  function 
**Limited  sensitivity 


TABLE  5 

Estimated  Sample  Sizes  Required  for  Achieving  a  Significant 
Load  Effect  for  Techniques  not  Demonstrating  Sensitivity 


Technique 

Estimated  Sample  Size 

SPARE  MENTAL  CAPACITY; 

Digit  Shadowing 

18 

Memory  Scanning 

>100 

Mental  Arithmetic 

25 

Time  Estimatipn  (Mean) 

53 

Time  Estimation  (Abs.  Error) 

>100 

Time  Estimation  (RMS  Error) 

85 

PHYSIOLOGICAL 

Pulse  Rate  Variability 

45 

Respiration  Rate 

15 

Pupil  Diameter 

>100 

Speech  Pattern  (D.  Shadow.) 

28 

Speech  Pattern  (M.  Arith.) 

>100 

EYE  BEHAVIOR 

Eye  Transition  Frequency 

42 

Eye  Blink  Frequency 

25 

PRIMARY  TASK 

Localizer  RMS  Error 

12 

Glide  Slope  RMS  Error 

41 
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Figure  1„  Mean  normalized  scores  for  the  Cooper-Harper  rating  scale  measure 
plotted  as  a  function  of  load. 
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Mean  normalized  scores  for  the  WCI/TE  rating  scale  measure  plotted 
as  a  function  of  loado 
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Figure  3»  Mean  normalized  scores  for  the  time  estimation  standard  deviation 
measure  plotted  as  a  function  of  load. 


LOAD 

Figure  4„'  Mean  normalized  scores  for  the  pulse  rate  mean  measure  plotted  as  a 
function  of  load. 
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Figure  5.  Mean  normalized  scores  for  the  control  movements  measure  plotted  as 
a  function  of  load. 
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SECTION  III 


TAILORING  FLIGHT  TESTS  TO  IDENTIFY 
PILOT  DYNAMICS  AND  TASK  PERFORMANCE 


An  Annotated  Bibliography  of 
Objective  Pilot  Performance  Measures 
by 

T.  R.  Mixon  &  W.  F.  Moroney 

Naval  Postgraduate  School 
Monterey,  CA 


INTRODUCTION 

The  measurement  of  pilot  performance  is  a  "sine  qua  non."  Without 
measurement  one  cannot  discriminate  success  from  failure,  progression 
from  regression.  Measurement  is  a  means  for  determining  where  one  is, 
where  one  has  been,  and  potentially  where  one  is  going.  S.S.  Stevens 
defined  measurement  as  "the  assignment  of  numerals  to  objects  or  events 
according  to  rules."  This  document  reviews  attempts  to  assign  numerals 
to  pilot  activities  according  to  rules.  Thus,  the  emphasis  is  on 
objective  performance  measures,  i.e.,  those  measures  which  can  be  obtained 
without  observer-system  interaction. 

Some  of  the  areas  on  which  pilot  performance  measurement  (PPM)  impacts 
are  listed  in  Figure  one.  Workload  is  only  one  of  those  areas,  yet  to 
some  extent  "Pilot  Workload  Measurement"  could  have  been  inserted  at  the 
center  of  the  figure.  Indeed  pilot  performance  and  workload  are  closely 
interviewed,  both  have  been  measured  and  manipulated  under  a  variety  of 
circumstances  ranging  from  mathematical  model  to  the  real  world.  As 
suggested  on  Figure  2,  these  are  advantages  and  disadvantages  associated 
with  gathering  data  under  each  circumstance. 

Buckout's  review  in  1962  was  the  last  comprehensive  examination  of 
the  pilot  performance  measurement  literature.  The  annotated  bibliography 
discused  in  this  paper  attempted  to:  (1)  gather  the  PPM  literature  written 
subsequent  to  1962  into  one  source  (2)  describe  the  scenarios  and  measures 
used  in  collecting  PPM  data;  and  (3)  summarize  the  major  premises  and 
findings  of  each  article. 
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A  variety  of  sources  including  computer  based  literature  searchs 
were  used  to  identify  candidate  articles.  Ultimately  all  referenced 
material  was  divided  into  three  catergories:  (1)  objective  pilot 
performance  measures;  (2)  subjective  pilot  performance  measures;  and 
(3)  general  analysis  and  review  articles.  Each  of  these  sections  are 
discussed  separately  below: 


I  OBJECTIVE  PILOT  PERFORMANCE  MEASURES 


A  total  of  189  different  reports  which  use  objective  pilot 
performance  measures  were  located.  For  each  report  the  following  data 
were  provided: 

1.  Standard  Bibliographies  Reference  Material :  Author(s),  Title,  Source(sl 

2.  Subjects.  Number  and  type  of  personnel  used  in  the  research. 

3.  Equipment.  Each  experiment  utilized  either  a  laboratory,  a  flight 
simulator,  or  an  actual  aircraft.  The  distinguishing  line  between 
"laboratory"  and  "flight  simulator"  was  not  always  clear;  usually 
if  a  full-sized  aircraft  cockpit  was  employed,  this  was  classified 
as  a  "simulation"  experiment.  When  more  than  one  form  of  equip¬ 
ment  was  utilized,  this  was  so  indicated. 

4.  Scenario.  A  broad  definition  of  airborne  flight  including  missions 
and  flight  segments. 

Measures .  Observed  parameters  or  variables  measured  in  the  objective 
sense  without  human  perceptions  or  judgements.  These  generally 
fell  into  six  classes;  (1)  physiological,  (2)  aircraft  systems,  (3) 
man-machine  system,  (4)  time,  (5)  frequency,  and  (6)  combined 
measures.  Mathematical  and  statistical  transformations  applied  to 
each  measure  are  included  in  parentheses  after  the  measure. 

6.  Summary,  A  capsulized  synopsis  of  the  purpose  of  the  experiment, 
experimental  conditions,  and  the  results.  Brevity  was  preferred 
over  repetitious  statements  of  facts.  No  attempt  was  made  to 
review  or  critique  a  document. 

Figure  3  is  a  sample  of  the  reporting  format.  The  articles  were  classified 
according  to  the  following  format: 
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Descriptor 

Article  Number 

Field  Conditions 

Fixed  Wing 

Rotary  Wing/VSTOL 

100-120 

120-149 

Simulator  Conditions 

Fixed  Wing 

Rotary  Wing/VSTOL 

200-276 

277-287 

Laboratory  Conditions 

300-319 

Combination  of  Field 

Conditions,  Simulator 

400-434 

and/or  Laboratory  Conditions 

After  all  the  literature  had  been  reviewed  188  different  measures  were 
identified.  Matrices,  which  divided  the  188  performance  measures 
into  six  main  classes:  physiological,  aircraft  systems,  man-machine 
systems,  time,  frequency,  and  measures  of  effectiveness/other  were  developed. 
By  using  these  matrices,  the  researcher  can  identify  articles  of  interests 
which  use  a  common  performance  measure. 


II  SUBJECTIVE  PILOT  PERFORMANCE  MEASURES 


A  small  sample  (N=34)  of  subjective  pilot  performance  measurement 
studies  were  included  to  contrast  the  pilot  rating  method  with  the 
objective  measures  reported  in  Articles  100-434.  In  addition  to  the 
bibliographic  data  this  section  contains  either  the  author's  original 
abstract,  modified  slightly  in  some  cases,  or  a  modification  of  the 
sources'  introductory  material . 

Ill  RELATED  ANALYSIS  AND  REVIEWS 


This  section  contains  articles  which  while  related  to  pilot 
performance  measurement  did  not  contain  objective  or  subjective 
data  but  rather,  addressed  related  issues  such  as  data  analysis, 
simulation,  transfer  of  training,  prediction  of  pilot  performance, 
math  models,  etc.  In  addition  to  the  bibliographic  data,  this 
section  contains  either  the  author's  original  abstract,  modified 
slightly  in  some  cases,  or  a  modification  of  the  source's  introductory 
material . 

INDEXES 


In  an  effort  of  this  size  retrieval  of  the  article(s)  of  interest 
is  critical.  Therefore,  in  addition  to  the  matrices  described  previously 
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the  following  indexes  were  developed:  Author,  Subject  Matter,  Scenario, 
Performance  Measurement,  Source,  and  Accession  Number. 

REPORT  AVAILABILITY 

The  annotated  bibliography  described  above  will  be  released  by  the 
Naval  Training  Equipment  Center  as  IH  330  in  March  of  1982.  A  limited 
distribution  annotated  bibliography,  based  on  a  survey  of  the  limited 
distribution  and  Department  of  Defense  classified  literature  will  be 
released  by  the  Naval  Postgraduate  School  in  the  Fall  of  1982  as 
NPS  55-81 -01  OPR . 

The  authors  hope  that  these  documents  will  (1)  provide  a  means  for 
integrating  the  PPM  literature,  and  (2)  serve  as  an  impetus  to  develop 
a  systematic  approach  to  PPM. 

ADDITIONAL  MATERIAL 

Readers  who  are  interested  in  PPM  as  a  means  of  evaluating  skilled 
operator  performance  might  wish  to  consult  Mixon  (1981). 
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Quantification  and 


Areas  on  which  Aircrew  Performance  Measurement  Impacts 


The 

Real 

World 


Observations  and 
Measurements  on 
the  Real  World 


h< — Field  Studies — H 


-Simulations,,  Models  and  Games- 
h - Laboratory  Experiments- 


MathematicaL^. 
Models  H 


High - FIDELITY  of  TEST  CONDITIONS - LOW 

HIGH - COST - - - LOW 

LOW - EASE  and  FLEXIBILITY  of  TESTING - HIGH 
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Figjre  2 


Conditions  under  which  PPM  and  -Workload  data  have  been  gathered 
and  some  associated  Trade-offs  (Modified  from  Chapanis  & 

Van  Cott,  1972) 


266.  KELLY,  Michael,  J.,  Wooldridge,  Lee,  Hennessy,  Robert  T. , 
Vreuls,  Donald,  Barnebey,  Steve  F.,  Cotton,  John  C.,  and 
Reed,  John  C. ,  Air  Combat  Maneuvering  Performance 
Measurement ,  Canyon  Research  Group,  Inc. ,  Westlake  Village, 
CA  91361,  Contract  No.  F33615-77-C-0079 ,  sponsored  by  Naval 
Training  Equipment  Center,  Orlando,  FL  32813,  and' Air  Force 
Human  Resources  Laboratory,  Brooks  Air  Force  Base,  TX  78235, 
NAVTRAEQUIPCEN  IH-315/AFHRC-TR-79-3 ,  September  1979,  142  pp. 
See  also  Proceedings  of  the  23rd  Annual  Meeting  of  the  Human 
Factors  Society,  1979,  p.  324-328. 

SUBJECTS:  30  fighter  pilots. 

EQUIPMENT:  Simulator  for  Air-to-Air  Combat  (SAAC) 

configured  as  an  F-4  aircraft. 

SCENARIO:  One-vs-one  air  combat  maneuvering. 

MEASURES:  Altitude  rate  (mean),  opponent  out  of  view 

(percentage  of  time  opponent  out  of  pilot's 
field  of  view)*  airspeed  (mean),  speedbrake 
(mean  deflection),  fuel  flow  (mean),  relative 
altitude  use  (ratio  of  altitude  standard 
deviations),  energy  management  index 
(function  of  remaining  fuel,  fuel  flow, 
airspeed  and  altitude),  offensive  time  (sight 
angle  less  than  60  degrees),  offensive  time 
with  advantage,  throttle  percentage  time 
(idle,  LO  MIL,  HI  MIL,  and  afterburner), 
heading  (root-mean-square  and  absolute 
average),  lead  time,  time  within  range,  roll 
rate,  maneuvering  rate  (roll  rate  times 
altitude  rate),  ACM  plane  of  action 
(composite  of  X,  Y  and  Z),  defensive  time, 
angle  of  attack  (percentage  of  time  greater 
than  28  units),  and  aircraft  kills 
(percentage  of  engagements  ending  in  an  AIM-9 
success,  gun  success,  ground  impact,  over-g 
or  fuel  exhaustion). 

SUMMARY:  The  goal  of  this  study  was  to  develop  a 

preliminary  measurement  structure  and 
measurement  set  for  an  automated  Air  Combat 
Maneuvering  Performance  Measurement  (ACMPM) 
system  which  could  be  implemented  on  the 
SAAC.  The  measurement  system  was  to  provide 
valid  and  diagnostic  performance  information 
in  real  time.  Using  multivariate  analysis,  a 
measurement  model  containing  13  variables 
accounted  for  51  percent  of  the  performance 
variance  and  was  able  to  discriminate  between 
pilots  of  high  and  low  skill  with  an  accuracy 
of  92.1  percent.  It  was  recommended  that 
further  analyses,  developmental  testing,  and 


Figure  3 

Sample  of  reports  included  in  annotated  bibliography 
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A  Study  of  TaBk  Difficulty  With  A  Subjective  Rating  Scale 
D.  W.  Repperger,  D.  B.  Rogers,  R.  E.  Van  Patten,  J.  Frazier 


Air  Force  Aerospace  Medical  Research  Laboratory 
Wright  Patterson  Air  Force  Base,  Ohio  45433 


SUMMARY 


The  results  of  two  experiments  are  discussed  which  relate  to  task  difficulty 
and  the  effects  of  environmental  stress  on  tracking  performance.  The  first 
experiment  involved  5  different  sum  of  sine  tracking  tasks  which  humans  tracked 
both  in  a  static  condition  and  under  a  5  Gz  acceleration  stress  condition.  The 
tasks  were  designed  in  such  a  manner  as  to  investigate  workload  measures  and  to 
compare  our  hypothetical  design  to  subjective  evaluations.  The  tasks  were 
required  to  satisfy  5  criteria  specified  in  mathematical  terms. 

The  second  experiment  involved  similar  environmental  stress  conditions  but 
in  this  case  the  tasks  were  constructed  from  deterministic  functions  with 
specially  designed  velocity  and  acceleration  profiles.  In  both  parts  of  this 
experiment,  subjective  evaluations  were  obtained  and  compared  to  the  assumption 
that  difficulty  is  related  to  magnitudes  of  velocity  and  acceleration  profiles 
of  the  target  tracking  task.  Phase  Plane  performance  analysis  was  conducted 
across  7  subjects  to  study  potential  measures  of  workload  or  tracking 
difficulty. 


INTRODUCTION 


In  the  study  of  manual  control  theory,  the  systematic  characterization  of 
task  difficulty  has  been  a  problem  of  considerable  interest  for  many  years.  An 
extensive  amount  of  work  has  been  done  in  this  area  and  a  variety  of  studies 
indicating  different  measures  related  to  workload  are  available  in  the  Human 
Factors  and  Psychological  literature.  In  the  engineering  literature,  the 
classical  paper  by  Cooper  (reference  l)  illustrates  the  motivation  for  such  a 
characterization  of  task  difficulty  -  a  subjective  rating  scale  for  human 
tracking.  The  extent  at  which  this  subjective  rating  scale  can  be  used  to  elict 
pilot  response  is  best  illustrated  in  reference  2  where  a  thorough  study  has 
been  done  to  investigate  and  pinpoint  the  exact  cause-effect  relationships 
between  pilot  subjective  ratings  and  handling  qualities  of  aircraft.  This  study 
uses  a  decision  tree  type  of  analysis  procedure  to  investigate  the  responses. 

At  the  Air  Force  Aerospace  Medical  Research  Laboratory,  it  is  of  interest  in 
our  research  program  to  develop  standard  tasks  or  levels  of  tracking  difficulty 
and  to  be  able  to  estimate  levels  of  difficulty  associated  with  human  tracking. 
Once  a  consistent  set  of  tasks  are  developed  which  provide  a  basis  or  standard 
for  tracking  behavior,  it  is  then  possible  to  more  closely  evaluate  the  effects 
of  stress  on  human  tracking  performance.  The  criteria  for  the  design  of  the 
tracking  tasks  must  be  such  that  each  task  i3  required  to  be  a  sensitive 
indicator  of  performance  change  (between  each  task  number)  and,  in  addition, 
the  requirement  is  made  that  the  task  is  to  be  sufficiently  sensitive  as  to 


•how  a  performance  decrement  between  the  atress-non  stress  condition. 

This  study  consisted  of  two  separate  experiments.  Both  parts  of  this 
Investigation  involved  human  tracking  for  target  forcing  functions  with 
different  acceleration  and  velocity  profiles.  It  was  desired  to  study  a 
critical  task  concept  (reference  3)  based  on  a  hierarchy  of  difficulty 
associated  with  the  different  target  forcing  functions.  This  approach  differs 
from  the  classical  critical  task  concept  considered  by  Jex,  et  al.  (reference 
4)  in  'which  the  controlled  element  would  have  dynamics  that  change.  In  our 
studies,  the  controlled  element  (figure  (l))  remained  the  same;  the  tracking 
tasks  varied  in  levels  of  difficulty  based  on  our  hypothesis  of  different 
velocity  and  acceleration  profiles  associated  with  each  target  forcing 
function.  The  motivation  for  this  work  was  due  to  an  interesting  paper  by 
Verplank  (reference  5)  in  which  he  equated  difficulty  and  stress  in  studying 
human  response  behavior  within  a  vigilance  paradigm. 


SYMBOLS 


f(t)  »  The  Target  Forcing  Function  Signal 

•(t)  -  The  Closed  Loop  Error  Signal 

x(t)  ■  The  Output  of  The  Plant  (Controlled  Element) 

B  •  Radius  in  the  f  versus  */  plane  ■  (1)^  +  (f)2 

|i  ■  Median  of  the  distribution  of  the  error  window  histogram 

•BUS  ■  Root  Mean  Square  error  score 

I  ■  mean  e^jis  value 

<T  ■  standard  deviation  of  epjis  value 

STj  “  The  deviation  of  a  difference  from  the  mean  of  the  differences. 
Md  *  Mean  of  the  n  differences  of  paired  observations. 

Probability 
■  t  test  statistic 
t  *  time 


METHOD 


Subjects  -  Seven  male  United  States  Air  Force  volunteers  participated  in  this 
experiment.  They  had  prior  training  in  both  the  G  type  of  stress  exposures  and 
manual  tracking  tasks. 


Design  of  The  Target  Tracking  Task  -  Part  I 


The  objective  of  this  study  was  to  develop  the  tracking  tasks  of  different 
levels  of  difficulty  and  to  study  their  ability  to  produce  performance 
decrements  between  tasks  (for  a  given  experimental  condition)  and  between 
experimental  conditions  (for  the  same  task).  For  the  first  part  of  this  study, 
it  was  decided  to  design  five  different  tasks  with  the  following  constraints: 

(1)  Each  tracking  task  will  be  zero  mean,  constant  variance,  sum  of  sines. 

(2)  Each  forcing  function  when  presented  as  a  replication  will  have  a  random 
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initial  phase  angle  for  each  frequency  component.  The  phase  angle  must  be  a 
prime  multiple  of  the  fundamental  frequency  and  not  a  linear  constant  multiple 
of  any  other  frequency  component. 

(3)  Each  forcing  function  will  have  a  random  phase  angle  between  each 
frequency. 

(4)  Due  to  human  physiological  exposure  limits  in  the  design  of  the 
acceleration  experiment,  the  length  of  each  task  was  set  at  15  seconds. 

(5)  The  amplitudes  of  all  the  sinusoids  are  scaled  so  such  that  they  all  have 
equal  power  and  produce  the  same  displacement  on  the  CRT  (display).  The  open 
loop  and  autopilot  runs  of  this  study  which  verify  this  fact  are  presented  in 
the  sequel. 

(6)  The  component  frequencies  of  the  sinuosids  are  "relatively  prime" 
multiples  of  a  fundamental  frequency. 

(7)  A  shift  in  frequency  content  is  required  so  that  ff^  >  ffj  is  true  if  i>j, 
i,j=1,2,..5  where  the  frequency  content  of  ff^  is  higher  than  that  of  ffj.  The 
procedure  for  obtaining  this  desired  result  is  discussed  in  reference  6. 

Using  a  measure  of  difficulty  denoted  as  R  (the  distance  in  the  target  phase 
plane  (figure  (2)))  where  R  satisfies: 

R2  -  (  f  )2  +  (  f‘)2  (0 

Then  table  I  illustrates  the  values  of  R  obtained  for  the  5  different  tasks 
chosen  in  Part  I  of  this  study. 


Table  I  Results  of  The  Open  Loop  and  Autopilot  Simulations 


Forcing  Function 
(or  Task) 

Number 

Open  Loop 
Error 

RMS  *  2351  • 

Autopilot 

Error 

RMS  *  2351 

R2(mean) 
For  The 
Forcing 
Function 

R2(s.d.) 
For  The 
Forcing 
Function 

m 

718.6 

477.8 

0.352 

0.290 

n 

718.6 

477.8 

0.662 

0.611 

#3 

718.6 

477.8 

1  .212 

1  .140 

U 

718.6 

477.8 

2.151 

_J^82 1 _ 

#5 

718.6 

477.8 

3.509 

3.2309 

In  this  table  the  results  of  the  open  loop  and  autopilot  runs  are  also 
displayed.  These  results  (columns  2  and  3)  illustrate  the  consistency  of  the 
normality  conditions  imposed  in  this  study  on  the  task  numbers. 


Design  of  The  Tracking  Task  -  Part  II 


In  this  design,  the  object  was  to  design  a  different  type  of  target 
forcing  funcion.  Figure  (3)  illustrates  the  shape  of  the  functions  used  in  this 
part  of  the  study.  In  this  case  the  objective  was  to  have  forcing  functions  of 
varying  difficulty.  The  assumption  is  that  the  radius  R  is  a  metric  of 
dispersion  about  the  origin  defined  by  equation  (l)  and  tasks  with  larger  R 
values  are  more  difficult  to  track.  The  design  of  the  function  in  figure  (3)  is 
a  result  of  the  need  to  have  target  tracking  tasks  that  varied  the  value  of  R 
as  a  function  of  time.  To  create  the  shape  of  the  diagram  in  figure  (3 ) »  the 
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following  exponential  functions  were  chosen  based  on  a  set  of  time  intervals 


Time  Interval(seconds) 

Function  Chosen 

(*1,  t2)  -  (0,5) 

fl(t)  -  a  e-(®-2,5)2/2a2ds 

(2a) 

(t2,  t5)  -  (5,10) 

f2(x)  ■  fj(5)+  (-ae“(8“7.5)2/2o2)ds 

(2b) 

(t3,  t4)  -  (10,15) 

f3(y)  -  f2(t0)*  ae-(^'2-5)2/2.2)dz 

(2c) 

(t4,  t5)  -  (15,20) 

f4(z)“^3(l5)+^Z(ae“(l'“17.5)2/2o2dt) 

(2d) 

Where: 

x-t-5 

(3a) 

y-t-10 

(3b) 

z-t-15 

(3c) 

yith  some  manipulation,  the  relationships  (2a-d)  and  (3a-c)  can  be  shown  to 
produce  the  trajectories  displayed  in  figure  (3)*  The  value  a  can  be  adjusted 
to  sweep  out  a  range  of  values.  Table  II  illustrates  the  values  chosen  for  part 
II  of  this  study: 

j 

Table  II  -  Forcing  Function  Design  For  Part  II 


FF  # 

a 

b 

1 

0.1 

.04 

2 

0.2 

.08 

3 

0.3 

.12 

4 

0.4 

.16 

5 

0.5 

o 

CM 

• 

Randomization  of  The  Presentation  of  The  Tasks 


Reference  6  describes  the  procedure  chosen  to  ensure  that  the  subjects  would 
not  know  the  order  of  presentation  of  the  five  different  tasks  at  any  time 
during  the  experiment.  Thi>3  was  true  for  Parts  I  and  II  for  both  the  static 
and  stress  portions  of  the  "experiment. 


Apparatus 


A  19-foot  arm  centrifuge  (figure  4)  was  used  to  establish  a  5  Gz  stress 
condition  for  the  subjects.  In  Air  Force  applications  this  acceleration  force 
is  in  the  z  direction  (down  the  spine  of  the  subject)  and  is  termed  Gz.  The 
centrifuge  rotated  at  an  angular  speed  of  27.5  RPM  with  the  cab  vectored  at  78 
degrees  about  a  line  in  the  z  axis  of  the  subject.  The  subjects  wore  standard 
Air  Force  helmets,  gloves,  and  an  Anti-G  Suit  with  a  G-valve.  The  Anti-G 
Suit-G-valve  delivers  a  specific  air  pressure  to  the  bladders  of  the  Anti-G 
Suit. 
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Training  Orientation  and  Data  Exposures 


During  this  training  orientation,  the  subjects  were  required  to  asymptote  to 
the  five  tracking  tasks  (performance  training)  and  also  to  acclimate  to  the  G 
stress  (physiological  adaptation).  In  the  final  design  of  this  experiment, 
each  day's  run  consisted  of  five  component  parts  or  phases  (figure  5 
illustrates  one  day’s  run  for  data  collection).  Phase  I  comprised  of  the 
presentation  of  the  five  tracking  tasks  in  the  static  condition  (no  stress). 
Phase  II  consisted  of  the  presentation  of  the  five  tracking  tasks  at  an 
acceleration  stress  level  of  5  Gz  with  a  20-second  preliminary  warm-up  run  at  4 
Gz.  After  the  five  exposures  at  5  Gz,  the  centrifuge  was  brought  to  a 
stationary  position  and  the  subject  again  performed  five  tracking  tasks 
presented  in  random  order  in  the  static  condition  (Phase  III).  Phase  IV  of  the 
daily  run  consisted  of  five  tracking  tasks  presented  in  random  order  again 
under  the  five  Gz  stress  as  in  Phase  II.  In  Phase  V  of  this  experiment,  the 
five  tasks  were  presented  in  the  static  mode.  Again,  as  with  all  the  previous 
tasks,  all  forcing  functions  were  presented  in  a  random  sequence.  Four  data 
days  were  collected  after  the  subject  progressed  satisfactorily  in  the 
indoctrination  period.  During  the  data  collection  phase  of  the  experiment,  the 
subject  never  experienced  more  than  300  seconds  per  day  of  5  Gz  exposure  nor 
more  than  two  daily  exposures  per  week.  After  the  4  data  days  were  collected, 
a  questionnaire  was  administered  on  the  fifth  day  with  the  subject  sitting  in 
the  centrifuge  but  with  no  machine  motion.  The  questionnaires  recorded 
subjective  ratings  of  the  task  difficulty  hierarchy. 


Questionnaire 


One  definition  of  workload  (reference  7),  indicates  that  it  is  a  function  of 
increased  performance  requirements  plus  additional  attention  requirements.  To 
get  a  true  subjective  evaluation,  it  was  necessary  to  ask  the  subjects  how  they 
rated  the  tracking  tasks.  On  the  last  day  of  the  experiment  the  subjects  were 
presented  25  tasks  in  random  order.  After  the  first  task,  each  subject  was 
asked  to  compare  the  task  he  was  presently  tracking  with  the  previous  one.  The 
subject  was  asked  whether  the  present  task  was  more  difficult,  less  difficult-, 
the  same,  or  not  possible  to  rate.  Thus  the  subject,  was  not  knowledgeable  as 
to  which  forcing  function  number  was  presented  and  would  only  give  relative 
ratings  between  tasks. 


RESULTS 


CDF  Performance  Results  From  The  Data 

As  discussed  previously,  after  the  5-day  indoctrination  period  the  seven 
subjects  tracked  until  they  trained  to  an  asymptotic  level  of  performance  for 
the  different  tracking  tasks.  One  criterion  used  to  define  asymptotic 
performance  is  that  on  three  consecutive  days,  the  RMS  performance  scores  do 
not  decrease  more  than  5/6  on  daily  exposures  of  25-50  static  presentations  of 
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the  five  random  targets  per  day.  After  this  level  was  reached,  the  subjects 
were  assumed  to  be  trained.  In  part  I  of  this  study,  the  first  question  to  be 
asked  concerns  the  adaptation  of  the  subjects  to  learning  the  tracking  task  and 
accclimation  to  G  levels. 

To  address  the  question  of  learning  and  adaptation  to  stress,  a  table  based 
on  error  scores  was  constructed  across  all  seven  subjects  and  four  replications 
Of  each  stress  condition.  Table  III  illustrates  these  results: 

Table  III  -  Stress  Data  T/a  Ratios  For  7  Subjects,  2  Replications/Day 


Day  1 

Day  2 

Day  3 

Day  4 

ran  -~J 

1  .9 

5.2 

2.9 

4.6 

wBmmm 

2.6 

4-9 

3-6 

4.0 

tt#3 

5-3 

4.5 

13.1 

10.6 

ff#4 

8.2 

10.0 

17.1 

13-5 

ff h 

10.1 

5-9 

rrw 

20.4 

If  any  trends  did  exist  in  the  data  runs,  due  either  to  further  performance 
training  (reduction  of  tracking  error)  or  possibly  to  further  acclimation  to  G 
stress  (physiological  adaptation) ,  they  would  be  shown  by  a  gradual  increase  in 
the  ratio  x/cr  across  a  row  for  a  given  forcing  function  number.  Since  there 
appear  to  be  no  apparent  trends  for  this  stress  acclimation,  it  is  assumed  that 
the  subjects  had  adjusted  to  a  steady  state  physiological  conditioning  and 
tracking  performance  level. 

The  next  question  to  be  addressed  here  is  whether  the  forcing  function 
number  was  correlated  with  measures  of  performance  degradation.  From  the  CDF 
figures  (similar  to  figure  6),  using  data  from  all  seven  subjects  (five 
replications),  it  was  desired  to  conduct  tests  to  investigate  if  a  significant 
performance  decrement  exists  dependent  on  forcing  function  number  for  both  the 
Static  or  the  stress  conditions.  The  following  statistical  test  would 
determine  this  effect: 

H0:  V 1+1  >  V  i  i*1  ,....4  (4a) 

versus  :  V  i+1  <  ji  i  i-1 , . .  .4  (4b) 

where  p  corresponds  to  the  0.5  line  on  the  CDF  in  figure  6.  This  figure  is 
Illustrated  here  to  show  how  the  median  point  p  is  obtained.  This  corresponds 
to  a  "median  window"  size  for  the  tracking  error  signal.  The  test  was 
conducted  for  both  the  static  data  and  the  stress  data.  The  results  using  a  t 
statistic  are  displayed  in  Table  IV: 


Table  IV 


Hypothesis  test 
on  values 

"t  for 

static  data 

p 

t  for 

stress  data 

P 

ffp  >  ffi 

5-174 

<•01 

1 1  .48 

<•01 

“  ff^  >  ff? 

18.39 

<.01 

19-91 

<.01 

ff4  >  ff3 

7.15 

<.01 

9*£>4 

<.01 

* — rrgmt - 

4.55 

<.01 

5-04 

<.01 

The  T  values  used  in  Table  IV  were  the  t  statistic  (2-tailed  test)  for 
correlated  data  (references  8  and  9)  which  satisfies: 
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t 


(5) 


Md7 


x2d 


»  n(n-l) 

where  Md  =  the  mean  of  the  n  differences  of  paired  observations  and  =  the 
deviation  of  a  difference  from  the  mean  of  the  differences.  This  test  is  for 
paired  samples;  they  are  not  independent  but  are  correlated  due  to  the  five 
replications  involved  with  all  seven  subjects  in  the  static  case,  and  four 
replications  with  the  stress  data.  The  results  from  Table  IV  indicate 
performance  decrements  correlated  with  the  forcing  function  number.  The 
performance  decrement  is  significant  at  an  .01  level  for  an  increase  of  forcing 
function  number  in  both  the  static  and  stressed  condition.  The  test  given  here 
corresponds  to  changes  in  medians  (i.e.,  the  0.5  point  on  the  CDF  curve). 

Using  the  CDF  method,  this  analysis  could  have  been  performed  for  any  window 
size  or  any  other  point  on  the  CDF  curve.  This  is  emphasized  here  because  in 
other  types  of  applications  it  may  be  desirable  to  look  at  a  specified  level  of 
the  CDF  function  (e.g.  CDF/  0.5)  or  at  a  specified  window  size.  Finally,  the 
tests  illustrated  here  hold  over  both  the  static  and  stress  conditions. 

Another  question  to  be  addressed  is  whether  the  effects  of  the 
physiological  stress  induce  a  performance  change  for  each  task  number.  Using 
the  data  from  the  seven  subjects  and  four  replications  of  the  stress  condition, 
Table  V  illustrates  the  effects  of  stress  on  tracking  performance. 


Table  V  Comparisons  of  Stress  vs  Static  Conditions 


Hypothesis  test 
on  P  values 

"i  for  this  test 

P 

ff1 

stress 

> 

ff^  static 

3-34 

<.05 

stress 

> 

ff?  static 

1 .54 

<.10 

stress 

> 

f f-^  static 

2.81 

<.05 

stress 

> 

ff4  static 

5-  S3 

<.01 

f±5 

stress 

fftj  static 

3-14 

<•05 

The  t  statistic  used  in  this  test  is  the  same  as  in  equation  5-  One  can  now 
see  the  impact  on  performance  degradation  as  noted  by  the  effect  of  stress  on 
tracking  in  Table  V. 

In  part  II  of  this  study,  it  was  desired  to  study  this  sensitivity  effect 
for  the  second  class  of  tracking  tasks.  Table  VI  illustrates  the  actual  error 
score  results  for  Part  II  as  well  as  the  equivalent  values  found  in  Part  Is 


Table  VI 


Part  I 

Part  II  ~| 

Stat.i  c 

Stress 

Static 

Stress  I 

ff  it 

x  mean 

x  s  .d . 

x  mean 

x  s.d 

x  mean 

x  s.d. 

x  mean 

x  s.d. 

i 

54.8 

26.4 

127.7 

29-6 

22.  er^ 

3.87 

>37773 

TTR 

2 

162.4 

48.7 

179.0 

33.2 

38.42 

6.71 

59.14 

12.02 

3 

283.9 

49.4 

306.3 

29.7 

54.32 

1 1  .40 

79.91 

14.88 

4 

388.6 

22.9 

414.7 

37.4 

106.32 

26.63 

121 .77 

34.89 

5 

624.5 

23.9 

649.5 

26.4 

223.09 

38.82 

239.96 

48.58 

The  results  of  the  statistical  tests  indicate  a  performance  decrement  under 
various  conditions.  The  subjective  data  from  the  questionnaire  are  presented 
next . 
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Results  From  The  Questionnaire 


In  the  questionnaire,  the  subjects  were  asked  to  compare  the  relative 
difficulty  of  the  task  they  were  presently  tracking  with  the  previous  task. 
Responses  of  "more  difficult",  "less  difficult",  "the  same",  or  "couldn't  tell" 
were  then  correlated  with  the  task  numbers  presented.  These  results  are 
displayed  in  table  VII  for  Parts  I  and  II  of  this  study: 


Table  VII  -  Correlation  of  Task  Numbers  with  Subjective  Responses 


Part  I  j 

|  Part  II  | 

Subject  # 

#  Correct/Total 

?  Correct 

#  Correct/Total 

?  Correct 

1 

24/25 

96/  \ 

21/25 

“84^ 

2 

23/25 

92? 

25/25 

100? 

3 

~  24725 

96? 

19/25 

~76? 

4 

25/25 

100? 

25/25 

100? 

5 

23/25 

92? 

25/25 

92? 

6 

25/25 “ 

92? 

24/25 

96?  ... 

7 

r~  25/25 

92? 

25/25 

100? 

The  subjects  also  commmented  that  as  they  were  presented  tasks  with  higher 
forcing  function  numbers,  the  tracking  tasks  required  more  attention.  This 
corresponds  to  the  description  of  workload  cited  earlier  in  which  higher 
performance  requirements  coupled  with  more  stringent  attention  requirements 
increase  workload. 


SUMMARY  AND  CONCLUSIONS 


This  study  used  sensitive  tracking  tasks  to  evaluate  performance 
degradation  under  acceleration  stress.  The  tasks  designed  here  had  to  satisfy 
certain  criteria.  First  they  had  to  be  zero  mean,  constant  variance,  sum  of 
sines.  Second,  open  loop  scores  for  all  five  tasks  had  to  be  identical.  Third, 
the  autopilot  runs  also  had  to  yield  a  consistent  score  for  all  five  tasks. 

When  the  human  was  tracking  these  tasks,  however,  a  performance  decrement  had 
to  be  observed  dependent  on  forcing  function  number  for  static  tracking.  In 
addition,  the  performance  decrement  had  to  occur  as  a  function  of  the 
experimental  conditions  stress  versus  non-stress  for  each  forcing  function 
number. 

At  the  conclusion  of  the  experiment  the  subjects  were  given  a 
questionnaire  to  rate  the  different  tasks.  Subjective  ratings  of  each  task  in 
order,  of  difficulty  were  necessary  in  order  to  verify  the  workload  definition 
used  here,  which  requires  both  a  performance  decrement  and  an  attention 
requirement  for  arranging  tracking  tasks  in  order  of  increasing  difficulty. 


*  Due  to  different  subject  pools  in  Parts  I  and  II  of  this  experiment,  subject 
#N  in  Part  I  may  not  be  the  same  person  as  Subject  #N  in  Part  II  (N=1,.,7). 
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PROJECT:  HEL  Cockpit  Lighting  Compatibility  (HELCLIC) 

PRINCIPAL  INVESTIGATOR:  MAJ  Thomas  L.  Frezell 


ABSTRACT 

1.  Background.  The  use  of  night  vision  goggles  is  considered 
imperative  to  developing  a  near  term  night  operational  capability 
for  US  Army  aviation  assets.  Cockpit  lighting  systems  in  current 
US  Army  fleet  helicopters  interact  adversely  with  night  vision 
goggles . 

The  US  Army  Human  Engineering  Laboratory  (HEL)  has  developed 
several  modifications  to  cockpit  lighting  systems  which  are  intended 
to  alleviate  the  present  goggle/lighting  incompatabilities .  These 
systems  have  been  tested  under  laboratory  conditions.  The  JUH-1H 
instrumented  helicopter  has  been  modified  with  two  of  the  most 
promising  candidates. 

2.  Objective.  Determine  the  effects  of  conventional  versus 
modified  cockpit  lighting  on  pilot  performance  while  flying  with 
night  vision  goggles.  (AN/AVS-6) 

3 .  Participants .  The  participants  in  this  experiment  will  be 
rated  aviators  from  both  the  HEL  and  Phillips  Army  Airfield.  The  N 
will  consist  of  from  five  to  eight  pilots. 

4.  Apparatus .  The  apparatus  will  consist  of: 

a.  JUH-1H  Instrumented  Helicopter 

b.  Night  Vision  Goggles 

c.  The  current  cockpit  lighting  and  modified  cockpit  lighting 
systems . 

5.  Procedures  and  Methodology.  Pilot  subjects  will  be  briefed  on 
the  general  nature  of  the  project  and  the  flight  maneuver  to  be 
performed.  The  flight  maneuvers  will  consist  of  a  360°  right 
hovering  turn  at  two  altitudes,  3  ft  and  50  ft.  These  maneuvers  will 
be  flown  for  all  test  conditions  (i.e.,  three  cockpit  lighting 
conditions  with  and  without  goggles.)  Initially,  each  subject  will 
complete  the  flight  maneuvers  under  the  three  lighting  conditions 
without  goggles  to,  preserve  their  dark  adaptation  and  conserve  flight 
time.  Dark  adaptation  will  be  accomplished  by  having  each  subject 
dark  adapt  with  red  goggles  for  thirty  (30)  minutes  prior  to 
initiation  of  flight  tests.  The  order  of  presentation  of  cockpit 
lighting  will  be  randomized  across  subjects  to  preclude  order 
effects . 

Each'  subject  will  be  asked  to  bring  the  aircraft  to  the 
appropriate  altitude  (3  ft  and  50  ft)  within  t 1  ft  (Aircrew  Training 
Manual  Standard)  and  complete  a  360°  right  hovering  turn  about 
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the  mast  of  the  helicopter  maintaining  altitude  within  +  ft,  drift 
not  to  exceed  2  ft  from  pivot  point  and  a  constant  rate  of  turn,  not 
to  exceed  90°  in  4  seconds. 

The  data  to  be  recorded  are  absolute  altitude  (radar)  Z_;  Doppler 
Position  Data  (x  &  y) ,  Flight  Control  Inputs  (cyclic,  pedals  and 
collective) .  Derived  measure  will  be  rate  of  turn  completed. 

All  maneuvers  to  be  performed  are  standard  flight  maneuvers. 

The  3  ft  hover  is  accomplished  during  every  flight;  while  the  10  ft 
and  50  ft  maneuvers  are  accomplished  prior  to  performing  OGE  (out  of 
ground  effect)  flight.  Each  subject  will  provide  three  sets  of 
maneuvers  for  each  altitude. 

6.  Experimental  Design.  The  experiment  will  be  conducted  as  a 
3x2x2  factorial  design  with  repeated  measures.  The  independent 
variables  will  be  (1)  Flight  Maneuvers;  (2)  Goggles/No  Goggles  and 
(3)  Cockpit  Lighting  Systems.  The  dependent  variables  will  be  (1) 
Hover  Position;  (2)  Turn  rate;  (3)  Flight  Control  Activity  and  (4) 
Absolute  Altitude  Variations.  Each  subject  will  have  three  trials 
per  test  condition. 

The  order  of  presentation  of  lighting  conditions  will  be 
randomized  across  subjects. 

7.  Data  Analysis.  The  flight  data  is  recorded  on  magnetic  tape  and 
consists  of:  the  altitude  above  ground,  the  position  over  ground, 
aircraft  heading  and  flight  control  activity.  The  control  activity 
consists  of  the  cyclic  latitude,  the  cyclic  longitude,  the 
collective  and  the  pedal  movements.  The  appropriate  sensors  are 
sampled  20  times  per  second  and  are  recorded  on  the  analog  tape  by 
the  helicopter  instrumentation  package  (reference,  TM  17-80)  .  The 
recorded  data  is  transferred  to  digital  tape  after  the  flight  and 
read  into  a  data  file  on  the  Cyber  7600  computing  facility  here  at 
Aberdeen  Proving  Ground,  Maryland. 

Event  markers  will  be  set  during  the  test  to  mark  the  start  and 
end  of  each  test  run  for  the  conditions  of  a  subject  at  a  given  test 
altitude,  cockpit  lighting,  goggles  usage,  and  replication.  The 
setting  of  these  toggle  switches  will  be  recorded  on  the  flight  data 
tape  and,  in  turn,  read  into  the  Cyber  data  file. 

The  mean  and  standard  deviation  shall  be  computed  from  the  Cyber 
data  file  for  each  flight  activity  of  each,  .test  run.  The  means  will 
be  used  as  cell  entries  in  a  3x2x2  random  block  Analysis  of  Variance 
with  repeated  measure  for  each  flight  activity.  The  analysis  shall 
be  for  the  independent  variable  of  altitude,  copkpit  lighting  and 
goggle  usage.  r,v:iyi 
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OVERVIEW  OF  WORK  IN  PROGRESS  ON 
NON-INTRDSIVE  ASSESSMENT  OF  PILOT  WORKLOAD  AND  PILOT  DYNAMICS 


INTRODUCTION 

The  purpose  of  this  paper  is  to  describe  one  general  approach  to  non- 
intrusive  assessment  of  pilot  workload  and  pilot  dynamics,  how  this 
assessment  is  carried  out,  and  recent  and  on-going  projects  which  involve 
its  application.  We  begin  with  a  general  closed-loop  formulation  of  pi¬ 
loting  technique  which  can  include  both  psychomotor  and  cognitive 
aspects .  Examples  are  given  showing  how  this  basic  closed-loop  structure 
can  be  applied  to  a  variety  of  flight  tasks  with  the  help  of  several  quan¬ 
titative  examples.  We  then  go  on  to  consider  the  major  objectives  of  work 
in  progress,  some  of  the  tools  and  concepts  used  in  non-intrusive  piloting 
technique  assessment,  and  a  number  of  the  important  lessons  learned  in 
applying  various  procedures.  Finally,  we  summarize  some  of  the  recent  and 
on-going  projects  involving  non-intrusive  piloting  technique  assessment 
procedures . 


BACKGROUND 


I:  l,  r,  ','fj  “i  I  O  Q  O  -1  :  1  -  :  ■ 

Historically  we  have  depended  heavily  upon  the  pilot  to  provide  us 
with  'direct  and  explicit  Information  concerning  manual  control  strategy, 

,  ra,  J  jyr  n  f>  r  h  aSO'jhbs  OgJ  8  i-:-  •  SV  '  r  <r  1 

perceptual  dynamics, -and  pilot  workload.  The  last  item  is  often  expressed 

]  T\iT )  h no v*  J>tsp  {asrjniJo  s-Bi .'.>>> on  **.o  k.ilu 

in  terms  of  a  pilot  opinion  rating.  Our  dependence  upon  pilots  to  assess 

,  .coh1.-‘?>Xss  noi'.'fE’njs'c'inoo  , ,  a.  s)  sysani  ; 

their  own  actions,  perceptions,  anddegree  of  stress,  however,  can  inter- 

c.yr\- -j  rv  **o  s n o t ^ o j- i « ?je.; jt?o \  \  r-'v;-  \  •  J’  1 

fere  with  ‘  what  it  is  that  -  -  we  are-  trying  to  measure.  For  example, 

•  B£)  t  %■  fioi  I?-.'.  L'  *  X- 

assessment  of  workload  using  a  Cooper-Harper  rating  first  requires  a  pilot 


with  special  trailing:  oi'ri qusirtg '  the- rating  scale .  This  evaluation  pilot 
may t not  have  sufficient  background  and  skills  for  the  particular  mission 
or  lair craft  type  ‘being  ‘evaluated. q  We  hAve  found  that  pilot  control  strat¬ 
egy  is  evert  more  difficult1  to'  determine  from' direct  interrogation  of  the 
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pilot.  While  it  is  relatively  easy  for  most  pilots  to  relate  to  the  no¬ 
tion  of  a  basic  closed-loop  structure  representing  piloting  technique  in 
performing  a  specific  task,  it  is  difficult  for  them  to  quantify  such  a 
loop  structure.  The  same  observations  are  true  for  perceptual  aspects  of 
the  pilot  dynamics.  Although  the  pilot  can  identify  aircraft  states  used 
in  accomplishing  a  particular  task,  it  may  not  be  possible  to  describe 
geometric  cue  patterns  or  sensations  actually  used  or  their  relative 
amount  of  use. 

It  is  therefore  useful  to  search  for  ways  of  measuring  and  quantifying 
pilot  dynamics  and  pilot  workload  connected  with  the  execution  of  given 
tasks  which  do  not  depend  on  direct  interrogation  of  the  pilot  but  rather 
upon  how  the  pilot  actually  performs  or  executes  tasks  in  a  flight 
environment . 


AN  OVERVIEW  OF  NON-INTRUSIVE  PILOT  ASSESSMENT 

Figure  1  shows  a  general  overview  of  manual  task  execution  in  terms  of 
the  pilot-vehicle  system  and  how  this  system  can  be  measured  in  order  to 
obtain  a  non-Intrusive  assessment  of  pilot  dynamics  and  pilot  workload. 
Task  execution  is  expressed  in  terms  of  a  general  feedback  control  sys¬ 
tem.  A  wide  spectrum  of  piloting  tasks  and  styles  of  task  execution  can 
be  cast  in  terms  of  this  same  general  topology .  In  addition  to  familiar 
continuous  psychomotor  tasks  (e.g.,  tracking  a  gunsight  pipper  error,  a 
localizer,  or  a  glide  slope)  we  can  also  address  discrete  maneuvers  (e.g., 
landing  flare;  airspeed,  altitude,  or  heading  change;  and  responding  to 
traffic  control  commands);  discrete  tasks  (e.g.,  configuration  selection, 
radio  tuning,  and  mode  switching);  communications  within  or  without  the 
cockpit;  and,  finally,  checklist  execution  procedures. 

Note  in  Fig.  1  that  the  interface  between  the  pilot  and  the  vehicle  is 
between  the  elements  of  control  strategy  and  the  vehicle  dynamics.  Per¬ 
ceptual  dynamics  are  often  regarded  as  part  of  the  pilot  dynamics  but 
could  be  alternatively  considered  as  part  of  the  vehicle  dynamics  under 
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KNOWNS  AND  ASSUMPTIONS 


519 


l 


Figure  1 .  ITon-Intrusive  Pilot  Assessment 


certain  conditions,  such  as  inclusion  of  simulator  motion  washouts  or 
visual  system  properties . 

Task  execution  in  general  consists  of  transforming  some  pilot  desire 
or  pilot  command  to  a  result  which  is  often  expressible  in  terms  of  a 
vehicle  state.  Depending  upon  the  difference  between  what  a  pilot  desires 
and  the  pilot's  perception  of  what  results,  a  control  strategy  is  called 
upon  to  define  a  control  action.  The  control  in  turn,  acting  through  the 
veh icle  dynamics,  yields  the  result  of  the  task  execution.  The  loop  is 
finally  completed  with  the  comparison  of  desire  with  result  via  the  per¬ 
ceptual  dynamics . 

In  general,  non-intrusive  estimation  procedures  make  use  of  the  pi¬ 
lot's  control  and  task  result ,  both  of  which  are  usually  easily  quantified 
and  measurable.  The  control  and  result  must,  however,  be  combined  with 
knowns  and  assumptions  concerning  the  various  elements  of  task  execution; 
i.e.,  we  might  choose  to  assume  a  form  of  control  strategy  and  judge  that 
we  know  the  vehicle  dynamics  and  approximate  perceptual  dynamics .  Skill¬ 
ful  utilization  of  such  information  often  permits  us  to  make  a  good 
estimate  of  pilot  dynamics.  Thus,  as  a  result  of  observing  how  a  pilot  is 
executing  a  task,  we  can  frequently  estimate  the  control  strategy  being 
used  and  the  perceptual  dynamics  involved  in  the  task . 

It  should  be  noted  that  we  could  also  identify  simultaneously  the 
vehicle  dynamics.  It  is  sometimes  desirable  to  do  so,  and  we  will  mention 
how  we  can  separately  measure  vehicle  dynamics  and  pilot  dynamics  while 
using  the  same  measurement  data.  The  final  step  is  to  take  the  estimated 
pilot  dynamics  and,  using  known  correlations  between  workload  and  pilot 
dynamics  such  as  listed  in  Table  1,  estimate  the  pilot  workload  or 
stress.  Thus  in  this  indirect  set  of  procedures  we  have  avoided  any  di¬ 
rect  interrogation  of  the  pilot  or  hindrance  of  piloting  activities. 

Figures  2  through  6  illustrate  a  number  of  examples  of  this  task  exe¬ 
cution  topology  for  which  some  quantification  has  been  derived  from 
simulator,  flight,  and  analysis.  Ihe  next  section  illustrates  some  of  the 
varieties  of  tasks  which  can  be  expressed  in  this  general  form;  however, 
pilot-vehicle  elements  are  assigned  other  more  appropriate  names. 
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CORRELATES  OF  WORKLOAD,  STRESS 
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Can  approach  pilot  workload  via  measurement  of  pilot  dynamics! 


DYNAMICS:  SWIFTNESS  AND  SURENESS 
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DYNAMICS:  RESPONSE  TIME  AND  SETTLING 
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DYNAMICS:  TIME  AND  CERTAINTY  OF  INTERPRETATION 
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DYNAMICS:  TIME  AND  SURENESS  OF  EXECUTION 
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and  Tuning  Radios 


DYNAMICS:  DURATION  OP  CHECK  AND  LIKELIHOOD  OF  SLIP-UP 
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SPECIFIC  TASK  ANALYSES 


Few  examples  exist  of  even  partially  quantified  pilot-vehicle  loop 
structures  for  full-task  analyses .  A  review  of  the  literature  shows  an 
emphasis  on  loop  structures  for  partial  task  analysis,  and  this  is  limited 
mainly  to  inner-loop  aspects .  Figures  7  through  13  (excerpted  from 
Ref.  1)  depict  some  of  those  few  cases  where  relatively  complete  outer 
loop  tasks  have  been  considered  and,  to  some  degree,  quantified.  Note 
that  each  task  is  described  first  verbally  and  then  in  terms  of  a  feedback 
control  block  diagram.  In  each  case  an  attempt  is  made  to  focus  only  on 
the  primary  task  loop.  In  the  interest  of  simplicity,  supporting  or  se¬ 
condary  loops  are  omitted  (e.g.,  for  a  speed  change  maneuver  only  the 
longitudinal  control  axis  is  addressed  -  supporting  pitch  attitude  con¬ 
trol  and  control  of  other  axes  are  not  shown  explicitly.)  In  each  example 
the  input  to  the  task  execution  block  diagram  is  the  pilot's  desired 
state,  and  the  output  is  the  actual  resulting  state.  In  all  cases  there 
is  an  explicit  control  strategy  shown,  but  an  explicit  perceptual  block  is 
included  only  for  the  helicopter  decelerating-approach-to-hover 
(Fig.  12)  .  Where  no  perceptual  block  is  shown,  perfect  pilot  perception 
of  a  state  is  assumed. 


OBJECTIVES  OF  NON-INTRUSIVE  PILOT  ASSESSMENT 

The  objective^  of  the  work  in  progress  are,  first,  to  assess  in  quan¬ 
titative  terras  t'he  control  strategy  and  the  perceptual  dynamics  for 
mission-oriented  flight  tasks  without  employing  measurement  procedures 
which  hinder  the  pilot;  i.e.,  we  are  looking  for  non- intrusive  measurement 
techniques .  Furthermore,  we  are  trying  to  develop  and  refine  non- 
intrusive  piloting  technique  measurement  procedures  for  routine  flight  and 
simulator  use.  The  utility  of  such  measurements  is  much  less  if  one  can¬ 
not  use  them  routinely  with  ease;  and  this  implies  that  one  has  to  look 
for  simple,  'fast  computational  procedures.  The  next  objective  is  to  clar¬ 
ify  and  quantify  already  established  relationships  between  non-intrusively 
measured  pilot-vehicle  dynamics  and  pilot  workload,  examples  of  which 
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Verbal  Description 


Adjust  bank  angle  by  applying  appropriate  pressure  to  the 
lateral  cyclic  stick.  Monitor  roll  response  using  either  the 
actual  horizon  or  the  cockpit  artificial  horizon. 


Feedback  Control  Description 

Stick  Force 


Piloting  Techniaue 

i -  ~  n 


Desired  Berik 


Stick  Deflection 


K»(i+y)e~ 

(1+TjS) 


Flight  Task  / 

Aileron  Deflection 
(Lateral  Swashplate) 

I 

Inner  Loop  / 

t  r 

1 

1 

Vehicle  louter  Loop 

K6 

TCI 

HI 

a 

rai 

inn 

!  1 
i  1 

■ 

_ i _ 

L  _ 


Adjustments  made 
on  the  basis  of 
other  roll  attitude 
response  parameters 
and  the  next 
exterior  flight  task 
(e»g*,  heading 
regulation) 


i - - - 


Control 
spring- damper 
properties 
including 
pilot’s  arm 


Actual  Bank 


St  ick-to-surf ace 
gearing 


Vehicle  roll 
response  propertie.s 
including  roll 
time  constant, 
spiral  divergence, 
dutch  roll,  and 
adverse  yaw 


Figure  7*  Bank  Angle  Regulation  Task  and  Piloting  Technique 
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Verbal  Description  (Ref.  2) 


When  a  deviation  from  the  desired  heading  occurs,  refer 
to  the  attitude  indicator  and  smoothly  establish  a  definite 
angle  of  bank  which  will  produce  a  suitable  rate  of  return. 
As  a  guide,  the  bank  attitude  change  on  the  attitude 
indicator  should  equal  the  heading  deviation  in  degrees  not 
to  exceed  30  deg.  (For  example,  if  the  heading  deviation  is 
10  deg,  then  10  deg  of  bank  would  produce  a  suitable  rate  of 
correction.)  This  guide  is  particularly  helpful  during 
instrument  approaches  at  relatively  slow  airspeeds.  At 
higher  true  airspeeds,  a  larger  angle  of  bank  may  be  required 
to  prevent  a  prolonged  correction.  A  correction  to  a  heading 
deviation  of  2  deg  to  5  deg  may  be  accomplished  by 
application  of  rudder. 


Feedback  Control  Description 


Commanded  Bank  Actual  Bank 


(closed  loop) 


T,s  1  sec 
<P 


Figure  8.  Heading  Regulation  Task  and  Piloting  Technique 
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Verbal  Description  (Ref.  2) 


To  enter  a  turn,  you  should  refer  to  the  attitude  indicator  while 
applying  smooth  and  coordinated  control  pressures  to  establish  the 
desired  angle  of  bank.  Bank  control  should  then  be  maintained 
throughout  the  turn  by  reference  to  the  attitude  indicator.  Cross¬ 
check  the  heading  indicator  or  turn  needle  to  determine  if  the  angle  of 
bank  is  satisfactory.  Trim  may  be  helpful  during  prolonged  turns  to 
assist  in  aircraft  control. 

To  roll  out  of  a  turn  on  a  desired  heading,  a  lead  point  must  be 
used.  The  amount  of  lead  required  depends  upon  the  amount  of  bank  used 
for  the  turn,  the  rate  the  aircraft  is  turning,  and  your  rollout 
rate.  As  a  guide,  a  lead  point  of  approximately  1/3  the  angle  of  bank 
may  be  used.  With  experience  and  practice  a  consistent  rate  of  rollout 
can  be  developed.  A  lead  point  can  then  be  accurately  estimated  for 
any  combination  of  angle  of  bank  and  rate  of  turn.  Hake  a  note  of  the 
rate  of  movement  of  the  heading  indicator  during  the  turn.  Estimate 
the  lead  required  by  comparing  this  rate  of  movement  with  the  angle  of 
bank  and  the  rate  of  rollout. 


Verbal  Description  (Ref.  3) 


In  a  typical  approach,  the  pitch  attitude  Is  3  to  4  deg 
for  the  DC-10.  As  speed  is  decreased  to  threshold  speed,  the 
pitch  attitude  will  increase  about  1  deg.  Landing  flare  is 
normally  initiated  at  approximately  30  to  40  ft  above  the 
runway  surface.  In  the  hypothetical  case,  if  the  airplane  is 
flared  to  a  zero  rate  of  descent  with  idle  thrust  and  a  speed 
of  just  under  threshold  at  touchdown,  the  pitch  attitude  will 
be  8  to  9  deg.  However,  with  a  typical  low  rate  of  descent 
at  touchdown,  the  pitch  attitude  will  normally  be  7  to 
8  deg.  Landing  with  a  50  flap  setting  decreases  the  pitch 
attitude  approximately  1  deg  over  that  for  35  flap.  There  is 
ample  tail  ground  clearance  for  a  normal  35  to  50  flap 
approach  and  landing,  even  with  the  main  landing  gear  struts 
fully  compressed  and  flat  tires.  Fuselage  contact  with  the 
runway  will  not  occur  until  approximately  14  deg  pitch 
attitude. 


Feedback  Control  Description  (Ref.  4) 


Ccunpens  ation 

H  ~  ,1  Rfiic 


Effective 
Pitch  Attitude 
Lag 

T  ~  . 5  to  1.2  sec 


sec  (DC-10) 


Figure  10.  Landing  Maneuver  Task  and  Piloting  Technique 
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Verbal  Description 


Effect  a  speed  change  Maneuver  by  simultaneously  chang¬ 
ing  pitch  attitude  and  offsetting  flight  path  upset  by  suit¬ 
able  use  of  collective  control.  Stabilize  on  the  desired  new 
speed  through  appropriate  use  of  pitch  attitude. 


Feedback  Control  Description 


Desired  Attitude  Actual  Attitude  Acceleration 


Crossfeed  to 
collective 
control 


Figure  1 1 .  Up-And-Away  Acceleration/Deceleration  Task 
and  Piloting  Technique 
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Verbal  Description 


Fly  a  descending,  decelerating  visual  approach 
terminating  in  a  AO  ft  hover  over  a  landing  pad.  Avoid 
abrupt  maneuvers.  No  approach  guidance  includes  only 
standard  aircraft  instruments  normally  used  for  visual 
approaches. 


Feedback  Control  Description  (Ref.  5) 


Commanded  speed  Actual  speed 


and  attitude  feedhacks 
Ty  &  A  sec 


Figure  12.  Decelerating  Approach  to  Hover  Task  and  Piloting  Technique 
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Verbal  Description 


Halt  all  forward  motion  with  respect  to  the  terrain  as 
rapidly  as  possible  without  ground  contact  (of  the  tail 
rotor)  or  excessive  increase  in  height. 


Feedback  Control  Description 


Commanded  Pitch  Actual  Pitch  Speed 


Tg  ~  .3  sec 

(involves  precise  control  of 
Though  not  shown,  the  piloting  attitude  and  is  critical  to 

technique  for  controlling  height  success  of  maneuver.) 

is  as  crucial  as  the  deceleration 
per  se . 


Figure  13.  NOE  Quickstop  Deceleration  Task  and  Piloting  Technique 
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relationships  are  given  in  Table  1.  Additionally,  we  are  trying  to  apply 
such  relationships  to  several  diverse  areas  including  evaluation  of  simu¬ 
lator  fidelity.  We  are  also  attempting  to  apply  non-intrusive  pilot 
measurements  to  improve  understanding  and  specification  of  handling  quali¬ 
ties;  to  improve  measurement  and  quantification  of  level  of  skill 
development  in  aircrew  training;  and,  finally,  to  improve  thereby  our 
understanding  of  the  sources  and  causes  of  aircrew  error,  which  compromise 
flight  safety. 


TOOLS  AND  CONCEPTS  BEING  APPLIED 

To  provide  a  better  idea  of  the  non-intrusive  techniques  that  we  are 
trying  to  develop  and  use,  we  would  like  to  present  a  list  of  tools  and 
concepts  which  are  used  in  non-intrusive  piloting  technique  assessment. 


Manual  Control  Theory 

First,  and  perhaps  most  important,  are  the  basic  ideas  of  manual  con¬ 
trol  theory  which  have  been  developed  over  a  number  of  years  to  represent 
observed  pilot  behavioral  changes  due  to  training  (learning  effects)  and 
variations  in  task  variables  (adaptation) .  It  is  important  to  note  that 
we  are  not  treating  the  pilot  behavior  as  somehow  mysterious,  random,  or 
irrational;  rather,  we  are  recognizing  that  the  pilot's  behavior  (in  terms 
of  control  strategy)  is  defined  in  terms  of  what  the  pilot  can  perceive 
about  the  state  of  the  aircraft  and  the  state  of  the  task  execution  and 
the  fact  that  the  pilot  is  comparing  what  he  desires  with  what  he  per¬ 
ceives  to  be  the  current  state  of  affairs  .  ffcst  of  these  underlying  ideas 
are  conveniently  summarized,  for  example,  in  Refs.  6  and  7  together  with 
their  extensive  bibliographies .  They  offer  a  practical  basis  for  de¬ 
scribing  the  important  characteristics  of  the  human  operator' s 
perceptually-centered  control  strategy,  a  concept  which  we  shall  define 
next . 
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Perceptually-Centered  Control  Strategy 


In  performing  a  multioperator/multiloop  control  task  (such  as  control¬ 
ling  traffic  or  flying  an  airplane)  each  human  operator' s  control  actions 
are  based  on  his  perception  of  the  situation  to  be  controlled.  The  per¬ 
ception  involves  a  selection  among  available  external  or  internal  cues  in 
more  than  one  sensory  modality,  and  the  verbal  or  manual  control  actions 
can  be  thought  of  as  deriving  from  cognitive  and  psychomotor  transforma¬ 
tions  of  these  perceptions .  It  is  this  selection  plus  transformation 
process  that  we  will  define  as  the  operator's  control  strategy.  In  es¬ 
sence,  the  selection/transformation  process  generates  the  control  output. 


Operational  manual  control  systems  are  typically  designed  to  require 

•Jf 

far  less  than  the  operator's  ordinary  limiting  capabilities  in  adaptation 
or  learning  of  control  strategy .  Performance  decrements  due  to  environ¬ 
mental  stress  are  therefore  seldom  observed  except  in  emergencies  when  the 
operator  is  near  his  limiting  performance.  Intrinsic  skill  development 
limits  in  control  strategy  can  be  measured  only  under  high  task-induced 
stress  conditions.  Particular  control  tasks  can  be  designed  to  emphasize 
particular  skill  factors  in  accordance  with  the  successive  organization  of 
perception  (SOP)  theory,  another  important  concept  first  set  forth  in 
Ref.  8  and  deserving  of  brief  elaboration  here. 


The  Successive  Organization  of  Perception 
Theory  for  Skill  Development 

The  SOP  theory  describes  the  human  operator's  synthesis,  by  means  of 
internal  organizational  modifications  derived  from  training  and  exper¬ 
ience,  of  progressive  arrangements  of  control  strategy  from  compensatory 
through  pursuit  to  precognitive  stages  -  or,  in  other  words,  a 


*Adaptation  changes  strategy  and  performance  in  a  new  environment, 
whereas  learning  or  skill  development  changes  strategy  and  performance  in 
successive  encounters  with  the  same  environment. 
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progression  from  behavior  patterns  which  exhibit  closed-loop,  to  combined 
open-  and  closed-loop,  to  purely  open-loop  properties. 

The  SOP  theory  leads  to  an  understanding  of  both  progressive  and  re¬ 
gressive  control  and  monitoring  behavior  during  training,  transfer, 
rehearsal,  and  stressful  operations.  It  can  also  be  associated  with  at 
least  one  concept  of  perceptual  motor  loading. 

In  our  concept  of  the  perceptual-motor  loading  components  of  pilot 
workload,  perceptual-motor  activity  is  carefully  defined  to  involve  only 
conscious  perceptions  and  actions.  It  is  handling  the  unpredictable 
(emergency)  or  unfamiliar  (lack  of  practice)  which  taxes  the  operator's 
workload  capacity.  In  this  context  the  three  stages  of  SOP  can  be  com¬ 
pared  on  a  perceptual-motor  load  (PML)  basis  . 


1.  Initial  stage  (compensatory  control)  .  The  early  phases 
of  learning  predominantly  involve  continuous,  conscious 
activity.  We  would,  therefore,  expect  a  high  PML  dur¬ 
ing  compensatory  operations . 

2.  Intermediate  stage  (pursuit  control)  .  The  operator 
makes  use  of  any  coherence,  pattern,  or  discrete  cues 
in  the  presented  stimuli.  Compensatory  control  ac¬ 
tivity,  although  present,  experimentally  shows  a 
regression.  This  implies  a  lower  sensory-motor  ac¬ 
tivity  level.  Therefore  we  would  expect  the  pursuit 
level  of  operation  to  have  a  lower  PML  than  the  compen¬ 
satory  stage,. 

3.  Final  stage  (precognitive  control) .  At  this  level  of 
skill,  most  of  the  operator's  output  consists  of  execu¬ 
tion  oJ:  stored  commands,  and  his  conscious  perceptual 
activity  is  mainly  concerned  with  decision-making  ac¬ 
tivity.  This  should  result  in  a  lower  PML  for  a  given 
control  task. 

Pilots  indicate  that  one  effect  of  noncurrency  is  a  general  roughness 
of  control  application  and  lack  of  precision.  This  causes  them  to  spend 
more  time  controlling  the  aircraft  (higher  workload) ,  which  leaves  less 
time  for  other  procedural  matters  involved  in  complex  tasks.  This  degra¬ 
dation  of  control  skill  corresponds  to  regression  on  the  SOP  control  skill 
scale  given  above.  Thus  lack  of  practice  on  a  skill  increases  the  percep¬ 
tual  motor  loading  of  that  skill,  resulting  in  less  workload  reserve 
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capacity  for  other  elements  of  a  complex  task.  It  is  apparent  that  lack 
of  practice  could  reduce  this  capacity  to  less  than  that  required  for 
carrying  out  the  remaining  elements  of  a  complex  task;  or  a  simple  emer¬ 
gency  could  arise  that  would  consume  additional  capacity,  thus  overloading 
the  pilot  and  resulting  in  degraded  system  performance,  if  not  failure. 

The  compensatory-pursuit-precognitive  pathways  structure  is  suitable 
to  represent  not  only  a  pilot  or  controller's  progression  to,  or  regres¬ 
sion  from,  higher  levels  of  Internal  cognitive  system  organization  in  a 
given  situation,  but  also  to  represent  grossly  the  possible  loop  struc¬ 
tures  when  different  levels  of  display  information  are  provided.  In  ad¬ 
dition,  the  process  can  even  describe  the  procedural  organization  and 
operating  discipline  among  individuals  on  the  flight  deck  or  within  the 
air  traffic  control  system.  Assessment  of  pilot  dynamics  and  workload 
thus  requires  special  attention  to  the  SOP  and  the  pilot  control  strategy 
forms  which  distinguish  each  stage  of  the  SOP.  For  a  discussion  of  this, 
the  reader  is  directed  to  Ref .  9 . 


Control  Strategy  Models 

Next  we  recognize  the  concept  and  the  distinction  between  continuous 
and  discrete  maneuver  control  strategy  models .  Ihe  basic  distinction 
between  a  continuous  model  and  a  discrete  maneuver  model  is  that  the  con¬ 
tinuous  model  is  a  basic  regulatory  or  tracking  task.  Ihe  discrete 
maneuver  is  something  that  the  pilot  does  once.  This  could  be  a  landing 
flare;  it  could  be  responding  to  a  traffic  controller  with  a  change  of 
heading,  a  change  of  altitude;  or  it  could  be  a  high-g  air  combat 
maneuver . 

Basic  and  applied  research  going  back  more  than  two  decades  has  demon¬ 
strated  that  the  human  operator's  control  strategy  can  be  fairly 
accurately  described  by  linear  differential  and/or  difference  equations. 
When  the  control  task  Is  time-varying,  such  as  controlling  an  aircraft  on 
final  approach,  then  the  coefficients  of  the  differential  equations  will, 
In  general,  also  be  time-varying.  The  differential  equations  describe  the 
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human's  operations  on  his  perceptions,  and  it  is  these  operations  that 
characterize  his  control  strategy.  A  convenient  (and  widely  accepted) 
method  of  expressing  these  differential  equations  is  to  use  the  Laplace 
transform  or  the  Fourier  transform. 

It  is  essential  that  the  analyst  choose  a  likely  candidate  for  a  con¬ 
trol  strategy  model  in  order  for  the  subsequently  estimated  parameters  of 
the  pilot's  or  controller's  describing  function  (a  particular  type  of 
Fourier  transform)  to  be  valid.  Indeed  it  is  the  analyst's  exercise  of 
a  priori  knowledge  and  experience  in  choosing  the  form  of  the  model  which 
permits  the  use  of  a  very  simple  and  efficient  mathematical  algorithm  for 
performing  the  estimation. 

Effective  values  for  pilot  parameters  including  gains,  lead  time  con¬ 
stants,  and  delays  can  be  approximated  from  the  identified  frequency 
response  of  the  pilot.  One  significant  feature  of  this  kind  of  informa¬ 
tion  is  that  where  the  pilot  is  anticipating  visually  perceived 
information,  i.e.,  leading  it,  he  is  faced  with  increasing  workload. 
Likewise  increased  values  of  pilot  gain  alone  will  reflect  increasing 
mental  concentration  on  the  control  task  itself.  In  addition,  the  pilot 
may  be  faced  with  an  increased  workload  requiring  division  of  his  atten¬ 
tion  betweeen  controlling  and  monitoring  tasks,  either  within  the  cockpit 
or  outside  the  cockpit  upon  acquisition  of  contact  flight  cues .  Decreased 
values  of  gain  and  increased  pilot  remnant,  sometimes  accompanied  by  in¬ 
creased  values  of  effective  neuromuscular  delay  may  reflect  division  of 
attention,  particularly  if  substantiated  by  accompanying  eye-point-of- 
regard  measurements . 

When  other  modalities  are  employed  to  reinforce  the  visual  modality, 
such  as  rotary  motion  cues  from  an  actual  aircraft,  certain  of  the  visual 
workload  requirements  can  be  reduced.  In  the  case  of  rotary  motions 
greater  than  semicircular  canal  threshold  levels,  the  low-frequency  visual 
lead  generation  requirements  are  reduced.  In  essence,  the  rotary  motion 
cues  permit  the  pilot  to  close  an  inner  loop  akin  to  that  of  a  rate 
gyro.  Hie  net  effect  is  to  reduce  the  effective  whole-task  time  delay  by 
about  0.15  sec  (which  also  happens  to  be  the  effective  neuromuscular  delay 
increment  required  to  develop  a  first-order  low-frequency  lead)  .  Thus  the 
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total  visual  workload  will  have  been  reduced  by  the  sensing  of  motion. 
One  could  say  that  the  total  workload  has  not  been  changed  because  what 
had  previously  been  done  with  the  visual  channel  was  now  accomplished  by 
the  visual  plus  motion  channels  acting  together.  However  the  motion  loops 
are  essentially  autonomic  nearly  reflexive  in  nature  so  the  addi¬ 
tional  modality  reduces  the  conscious  workload  as  reflected  by  pilot 
ratings .  The  reinforcing  use  of  aural  and  other  modalities  also  has  a 
qualitatively  similar  effect. 


Alternative  Dona ins  for  Identification  of 
The  Human  Operator's  Control  Strategy 

Another  important  tool  for  identifying  control  strategy  is  simply 
depicting  the  pilot-vehicle-task  behavior  in  a  variety  of  domains:  time 
domain,  frequency  domain,  and  the  phase-plane  domain.  Each  provides  a 
certain  kind  of  insight  into  how  the  pilot  is  operating  and  the  perceptual 
dynamics  involved.  Both  the  time  and  frequency  domains  enjoy  wide  use. 
The  phase-plane  involves  suppression  of  the  time  domain  and  shows  the 
direct  relationship  between  one  state  variable  or  control  with  another 
state  variable  or  control.  The  classical  kind  of  phase-plane  diagram  is  a 
plot  of  a  velocity  versus  a  displacement.  We  extend  the  idea  of  the  phase 
plane  to  include  a  plot  of,  for  example,  control  displacement  versus  a 
variable  being  controlled. 


Sunning  Least  Squares  Estimation  Methods 

One  final,  very  important  computational  tool  or  technique  is  the  use 
of  running  least-squares  estimation  methods.  These  methods  can  provide  us 
with  a  basic  automatic  parameter  identification  capability  but  at  very  low 
cost  in  terms  of  software  complexity,  computer  speed,  or  computer  capa¬ 
city;  and  they  permit  on-line  results  . 
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The  non-intrusive  techniques  employed  for  identification  of  the  human 
operator's  control  strategy  from  flight  and  simulation  test  data  can  be 
classified  as  sampled  data  correlation  in  both  the  time  domain  and  the 
phase  plane  using  a  least  squares  criterion  with  existing  inputs .  A  run- 
ning  multiple  linear  regression  technique  called  NIPIP  has  been  used  to 
estimate  the  human  operator's  control  strategy  on-line  in  real  time  as 
well  as  after  completion  of  the  testing.  A  "sliding”  time  window  is  used 
to  refresh  the  data;  thus  time-varying  control  strategy  can  be  identi¬ 
fied.  Sampled  data  correlation  in  the  phase  plane  has  also  been  employed 
successfully . 


SOME  LESSONS  LEARNED 

We  would  now  like  to  mention  some  of  the  important  lessons  learned  in 
applying  non-intrusive  measurement  procedures.  First,  we  believe  that  it 
is  crucial  to  exploit  fully  simple,  easily  visualized  mathematical  models 
before  increasing  model  complexity.  The  pilot-vehicle  should  be  studied 
one  loop  or  one  axis  at  a  time.  Frequency  partitioning  of  tasks  is  highly 
recommended.  We  have  also  found  that  it  is  prudent  to  rely  heavily  on 
first-principles  models  of  the  vehicle,  the  pilot,  and  the  task.  We  take 
advantage  of  the  principles  of  physics,  the  theory  of  manual  control,  the 
rules  for  automatic  control  design  and  adjustment  if  necessary  (e  .g . , 
Ref.  10),  and  the  geometrical  relationships  which  describe  the  world  which 
the  pilot  perceives  and  the  flight  paths  which  the  aircraft  flies.  These 
models  provide  the  analyst  with  the  basic  insight  for  adopting  increased 
complexity  if  that  is  necessary.  We  have  generally  found,  however,  that, 
after  studying  behavior  in  terms  of  simple  mathematical  models,  we  some¬ 
times  are  able  to  recast  our  results  in  even  simpler,  more  cogent  terms. 
In  contrast,  complex  computer  analysis  routines  and  reliance  on  large- 
scale  computers  sometimes  actually  reduce  the  number  of  things  which  can 


*Non-Intrusive  Pilot  Identification  Procedure. 
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be  analyzed  and  frequently  tend  to  obscure  the  basic  insights  and  under¬ 
standing  of  events . 

In  order  to  understand  better  the  results  from  the  measurement  proce¬ 
dures,  it  is  helpful,  if  possible,  to  work  in  the  running  time  domain; 
that  is,  to  observe  the  pilot  estimation  results  as  they  are  happening  and 
to  permit  non-stationarity  in  these  results.  It  is  important,  for  exam¬ 
ple,  to  allow  for  alterations  in  pilot  control  strategy  over  the  period  of 
a  landing  approach .  It  is  therefore  important  to  have  a  computational 
procedure  which  is  fast  enough  to  identify  non-stationary  piloting  tech¬ 
nique  .  Long-term  averaging  of  pilot  measurements  is  valid  only  when  one 
is  sure  that  the  flight  task  is  not  changing  in  the  long  term  and  that  the 
pilot  has  not  changed  the  piloting  technique  due  to  environmental 
disturbances . 

It  Is  especially  important  when  interpreting  results  to  consider  the 
various  perspectives  of  the  piloting,  engineering,  psychological,  and 
training  disciplines .  No  single  one  of  these  disciplines  dominates  as¬ 
sessments  that  are  being  made  of  the  pilot  dynamics  or  the  pilot 
workload.  At  the  same  time,  there  are  perspectives,  technical  approaches, 
and  bodies  of  literature  within  each  discipline  which  may  trigger  addi¬ 
tional  insight. 

Next  we  have  found  that  it  is  important  to  treat  mission-oriented 
flight  tasks  as  opposed  to  the  more  artificial  and  contrived  tasks  which 
may  be  conducted  in  a  laboratory  environment.  There  is  no  question  that 
laboratory  results  have  been  invaluable  in  establishing  basic  ideas  of 
manual  control  theory  and  relationships  between  pilot  dynamics  and  pilot 
workload,  but  laboratory  test  scenarios  have  limited  applicability.  MLs- 
sion-oriented  flight  tasks  require  that  we  deal  in  a  realistic  context, 
with  multiple  control  loops  and  with  the  proper  mix  of  cognitive  and  psy¬ 
chomotor  tasks.  In  so  doing,  however,  we  are  compelled  to  develop  and 
apply  non-intrusive  measurement  techniques  . 

Finally,  we  have  found  that  it  Is  important  to  be  skeptical  of  ex¬ 
trapolating  simulator  results  to  flight.  For  a  given  task,  and  especially 
for  critical,  difficult  tasks,  one  should  attempt  to  start  with  flight 
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data  in  order  to  validate  a  simulator.  Even  on  costly,  very  sophisticated 
large  amplitude  motion  simulators,  we  have  found  that  in  performing  such 
flight  tasks  as  landing  flare  or  low-altitude  helicopter  maneuvering,  the 
pilot  does  not  necessarily  fly  the  simulator  with  the  same  strategy  as  the 
pilot,  in  fact,  flies  the  real  aircraft.  Measurements  of  control  strategy 
and  perceptual  dynamics,  which  have  been  made  non-intrusively,  bear  this 
out . 


RECENT  AND  ONGOING  PROJECTS 

We  conclude  by  mentioning  some  recent  and  on-going  projects  which 
involve  non-intrusive  piloting  technique  measurement  procedures .  These 
projects  have  spanned  several  areas  and  have  included  various  applications 
for  measurement  of  pilot  control  strategy. 


Evaluation  of  Powered-Lift  STOL 
Safety  Margin  Displays 

One  general  application  has  been  the  documentation  of  how  pilots  use 
special  displays,  for  example,  use  of  a  safety  margin  display  for  the 
operation  of  a  powered-lift  STOL  aircraft  (Ref.  11).  This  work  was  per¬ 
formed  for  NASA  and  involved  use  of  a  fixed-base  simulation  of  the 
Augmentor  Wing  Jet  STOL  Research  Aircraft.  Non--intrusive  measurements 
revealed  tight  tracking  for  one  display  configuration  but  relatively  loose 
tracking  for  another.  The  latter  configuration  produced  worse  pilot  rat¬ 
ings,  a  predictable  result  with  knowledge  of  the  system  dynamics  and  the 
piloting  technique  of  the  pilot. 
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Identification  of  Pilot  Control  Strategy  in 
the  Use  of  Head-Up  Displays  for  CTOL  Aircraft 

NIPIP  was  used  to  identify  pilot  control  strategy  online  in  real  time 
during  a  ground-based  simulation  of  two  competing  concepts  for  head-up 
displays  (HUD)  and  a  head-down  attitude  director  indicator  for  use  in 
conventional  takeoff  and  landing  (CTOL)  aircraft  (Ref.  12  and  13).  Eleven 
professional  airline  captains  participated.  Data  for  three  of  the  pilots 
were  selected  for  more  detailed  comparison  and  discussion. 

The  "flight  path"  HUD  was  flown  with  a  higher  bandwidth  (or  crossover 
frequency)  and  lower  phase  margin  than  was  the  "flight  director"  HUD.  The 
"flight  director"  HUD  provided  a  fixed  aircraft  symbol  and  a  moving  dot  to 
display  the  vertical  and  lateral  guidance  director  commands  together  with 
a  numerical  display  of  speed.  Other  status  information  had  to  be  derived 
by  the  pilot  from  the  external  visual  field.  The  "flight  path”  HUD  of¬ 
fered  a  conformal  symbolic  display  of  the  direction  of  the  aircraft's 
flight  path,  suitably  compensated  with  lead  equalization,  the  reference 
glide  slope  angle,  and  an  angle  proportional  to  the  glide  slope  error.  In 
addition,  the  pilot  could  perceive  an  effective  director  command  :Ln  the 
symbology  of  the  "flight  path"  HUD  by  observing  the  angular  difference 
between  compensated  aircraft  flight  path  and  glide  slope  angle.  The 
"flight  path"  HUD  also  provided  an  analog  "fast'-slow"  speed  signal.  Both 
HUDs  incorporated  artificial  horizons. 

Examination  of  the  describing  function  data  revealed  that  the  pilots 
were  able  to  fly  the  head-up  displays  with  a  fairly  high  gain  and  use  lead 
compensation  in  their  control  strategy,  but  the  describing  functions  for 
the  head-down  display  exhibited  lag  compensation  and  a  low  bandwidth. 

The  pilot's  describing  functions  exhibited  a  trend  in  time-varying 
behavior  when  using  the  "flight  director"  HUD  and  head-down  display  but  to 
a  much  less  degree  when  using  the  "flight  path"  HUD.  Usually  the  pilots 
increased  their  gain  and  decreased  their  control  latency  as  they  came 
closer  to  the  minimum  decision  altitude . 
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NIPIP  was  also  able  to  detect  control  strategy  errors  with  the  head- 
down  display  and  the  "flight  path”  HUD.  These  errors  were  due  to  control 
reversals  with  the  head-down  flight  director  and  to  accidentally  using  the 
wrong  element  in  the  display  with  the  "flight  path"  HUD.  After  "breaking 
out"  (i.e.,  beneath  the  cloud  cover),  the  pilot  apparently  became  confused 
as  to  which  symbol  to  track  in  the  display.  This  caused  an  instability  in 
the  flight  director  loop.  This  sudden  change  in  control  strategy  caused  a 
sharp  decrease  in  the  amplitude  and  phase  of  the  pilot  describing  function 
and  corresponding  decreases  in  crossover  frequency  and  phase  margin.  The 
unstable  condition  lasted  for  only  a  few  seconds,  but  NIPIP  identified  the 
control  strategy  error  before  the  pilot  was  aware  of  it  and  applied  the 
proper  control  action. 


Design  of  a  Simulator  to  Study  Human  Error 

A  study  was  conducted  for  the  Man-ffechine  Systems  Division  of  NASA 
Ames  Research  Center  which  focused  on  human  error.  The  study  resulted  in 
the  description  of  a  theory  of  human  error  (Ref.  9),  how  human  error  might 
be  objectively  measured  (Ref.  14),  and  recommendations  for  the  design  of  a 
simulator  facility  for  making  such  measurements  (Ref.  15).  The  role  of 
non-intrusive  measurements  was  considered  essential  in  this  application. 


A  Comparison  of  Piloting  Technique  Made  in  an 
Airline  Training  Simulator  and  in  Actual  Flight 

An  analysis  of  pilot  control  strategy,  both  from  an  airline  training 
simulator  and  an  actual  DC-10,  was  performed  for  the  landing  maneuver 
(Ref.  4).  An  emphasis  was  placed  on  identifying  useful  metrics,  quanti¬ 
fying  piloting  technique,  and  defining  simulator  fidelity.  On  the  basis 
of  DC-10  flight  and  simulator  measurements  recorded  in  about  200  landings 

by  32  pilots  - 13  flight-trained  and  the  remainder  simulator-trained - a 

revised  model  of  the  landing  flare  was  hypothesized  which  accounted  for 
reduction  of  sink  rate  and  preference  for  touchdown  point  along  the 
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runway.  The  flare  maneuver  and  touchdown  point  adjustment  could  be  de¬ 
scribed  by  a  pitch  attitude  command  pilot  guidance  law  consisting  of 
altitude  and  vertical  velocity  feedbacks.  The  pilot  gains  which  were 
identified  directly  from  the  flight  and  simulator  data  showed  that  the 
flare  was  being  executed  differently  in  each  medium.  In  flight  most  of 
the  subject  pilots  exhibited  a  significant  vertical  velocity  feedback 
which  was  essential  for  well-controlled  sink  rate  reduction  at  the  desired 
level  of  response  (bandwidth)  .  In  the  simulator,  however,  the  vertical 
velocity  feedback  appeared  ineffectual  and  led  to  substantially  inferior 
landing  performance.  The  absence  of  the  vertical  velocity  feedback  im¬ 
plied  a  simulator  fidelity  problem  with  a  potential  impact  on  pilot 
workload,  and  several  specific  possibilities  were  discussed.  The  pilot 
model  of  the  maneuver  provided  insight  into  which  aircraft  types  could  be 
simulated  without  incurring  the  apparent  fidelity  limitation  encountered 
in  this  case. 


Identification  of  Pilot  Control  Strategy  in 
Simulated  Approaches  to  an  Aircraft  Carrier 

An  issue  of  concern  is  the  influence  of  pilot -generated  noise  on  the 
validity  of  pilot  control  strategy  identified  from  test  data.  One  recent 
study  resulted  in  predicting  results  of  a  simulation  of  the  carrier  air¬ 
craft  pilot's  approach  control  strategy  in  the  presence  of  pilot  remnant 
(Ref.  16).  The  aircraft  dynamics  and  the  turbulence  environment  were 
representative  of  a  (T-2C)  trainer-type  aircraft.  NIPIP  was  used  to  iden¬ 
tify  the  pilot's  control  strategy  required  by  this  highly-coupled, 
multiloop  control  task.  Of  particular  interest  was  whether  the  pilot 
learns  to  adopt  a  workload-reducing  pursuit  crossfeed  of  throttle-to-pitch 
attitude  command  which  enables  him  to  achieve  adequate  approach  preci¬ 
sion.  Ihe  results  were  presented  in  terms  of  frequency  responses  of  the 
individual  elements  of  the  pilot's  control  strategy.  It  was  found  that 
NIPIP  can  identify  the  pilot's  describing  functions  even  in  the  presence 
of  rms  pilot  remnant  amounting  to  25  percent  of  the  observed  rms  states 
when  no  remnant  is  present.  The  next  step  is  to  apply  NIPIP  to  a  real 
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time,  piloted  simulation  of  the  same  control  task.  This  is  planned  for 
the  Visual  Technology  Research  Simulator  at  the '  Naval  Training  Equipment 
Center  in  Orlando,  Florida. 


Determination  of  Motion  and  Visual  Systems 
Requirements  for  Army  Training  Simulators 

A  study  was  conducted  for  the  Army  Research  Institute  directed  at 
identifying  simulator  fidelity  requirements  for  training  (Ref.  1).  Both 
rotary-  and  fixed-wing  Army  missions  were  considered.  The  result  of  the 
study  was  the  description  of  a  systematic  method  for  approaching  simulator 
fidelity  requirements  which  included  direct  measurement  and  cataloging  of 
the  pilot  control  strategy  and  perceptual  dynamics  involved  in  each  train¬ 
ing  objective  (piloting  task)  .  Such  measurements  would  be  required  both 
in  flight  and  in  existing  simulator  facilities.  An  analytic  approach  was 
then  described  which,  based  on  data  obtained,  could  be  used  to  predict  the 
utility  of  future  systems  or  to  set  system  requirements.  Non-intrusive 
pilot  measurements  were  thus  considered  necessary  for  exploring  how  pilots 
operate  the  actual  aircraft  as  well  as  how  they  operate  the  training 
simulator  equipment . 


Design  and  Operation  of  a  New  Navy 
Advanced  Trainer 

A  study  was  conducted  for  the  Northrop/Vought  VTXTS  design  team  which 
involved  non-intrusive  pilot  measurements  for  evaluating  simulator  hard¬ 
ware  (Ref.  17).  A  carrier  approach  task  was  flown;  and  the  pilot  control 
strategy  was  measured  relative  to  vertical  flight  path  control,  pitch 
attitude  control,  and  throttle-to-attitude  crossfeed.  Several  pilots  with 
varying  backgrounds  and  levels  of  skill  were  used,  and  it  was  possible  to 
distinguish  these  in  the  control  strategy  measurements. 
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Research  Simulator  Fidelity  Using  Flight  and 
Simulator  Measurements  of  Control  Strategy 


Currently  the  fidelity  of  a  UH-60  Black  Hawk  helicopter  simulation  at 
NASA  Ames  Research  Center  is  being  evaluated  (Ref.  18).  Ihis.  is  being 
accomplished  using  pilot  control  strategy  measurements  from  actual  UH-60 
flights  for  a  variety  of  Army  flight  tasks  including  the  NOE  regime. 
Corresponding  control  strategy  measurements  will  be  performed  on  the  Ames 
Research  Center  Vertical  Motion  Simulator  (VMS)  in  order  to  find  dif¬ 
ferences  in  the  use  of  essential  motion  and  visual  cues  and,  thereby,  to 
infer  simulator  fidelity.  Ihis  work  will  also  result  in  an  initial  cata¬ 
loging  of  several  basic  flight  maneuvers  in  terms  of  pilot  control 
strategy  and  the  overall  closed-loop  dynamics  of  each  task. 


CONCLUSION 

Non-intrusive  assessment  of  pilot  dynamics  and  pilot  workload  has 
become  increasingly  more  important  as  we  begin  to  focus  on  real-world 
piloting  tasks  in  actual  flight  rather  than  in  simulated  flight.  teneu- 
vers  such  as  landing  flare,  tracking  tasks  such  as  the  carrier  approach, 
and  operating  environments  such  as  Army  NOE  often  preclude  intrusive  pilot 
measurements  or  contrived  laboratory  experiments .  Knowledge  of  the  pilot¬ 
ing  technique  involved  in  such  tasks  or  environments  reveals  much  in  terms 
of  essential  cues  for  simulation,  handling  qualities  requirements,  and 
training  needs . 

The  recent  and  on-going  studies  mentioned  herein  indicate  the  wide 
spectrum  of  applications  already  considered  for  non-intrusive  measure¬ 
ments.  During  these  same  projects,  much  has  been  acconplished  in 
developing  and  using  measurement  techniques.  It  has  been  important  to 
exploit  many  different  tools  and  concepts  within  the  areas  of  manual  con¬ 
trol  theory,  vehicle  dynamics,  control  system  analysis,  and  numerical 
analysis.  In  so  doing,  however,  effective  management  of  analysis  tech¬ 
niques  is  crucial.  This  has  tended  to  force  an  emphasis  on  relatively 
simple  modeling  and  measurement  procedures.  At  the  same  time. 


548 


mathematical  simplicity  has  proved  highly  effective  in  gaining  understand¬ 
ing  and  in  finding  where  increased  complexity  is  needed. 
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ABSTRACT 

The  Optimal  Control  Model  for  the  Human  Operator  is  reviewed. 
Underlying  concepts  are  presented  and  the  model  structure  is 
described.  Special  emphasis  is  given  to  the  treatment  of 
attentional  workload,  including  the  relationship  between  workload 
and  closed-loop  performance.  Validating  experimental  results  are 
presented,  and  both  predictive  and  diagnostic  applications  of  the 
model  are  reviewed.  Areas  of  further  model  development  and 
application  are  summarized. 


INTRODUCTION 

The  Optimal  Control  Model  (OCM)  for  the  human  operator  is 
based  on  the  assumption  that  the  well-motivated,  well-trained  human 
operator  will  act  in  a  near  optimal  manner  subject  to  the 
operator's  internal  limitations  and  understanding  of  the  task. 
This  assumption  is  consistent  with  notions  of  human  response 
behavior  discussed  in  the  psychological  literature.  What 
differentiates  the  OCM  from  other  models  of  the  human  operator  are 
the  methods  used  to  represent  human  limitations,  the  inclusion  in 
the  model  of  elements  that  compensate  optimally  for  these 
limitations,  and  the  extensive  use  of  state-space  concepts  and  the 
techniques  of  modern  control  theory. 

Clearly,  if  the  basic  optimality  assumption  is  to  yield  good 
results,  it  is  necessary  to  have  reliable,  accurate  and  meaningful 
models  for  human  limitations.  Insofar  as  possible,  these  models 
(or  their  parameters)  should  reflect  intrinsic  human  limitations  or 
should  depend  primarily  on  the  interaction  of  the  operator  with  the 
environment  and  not  on  the  specifics  of  the  control  task.  It  is 
also  desirable  that  the  description  of  human  limitations  involve  as 
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few  parameters  as  possible  and  that  it  be  commensurate  with  the 
modern  control  system  framework  that  is  being  employed.  These 
principles  have  guided  the  development  of  the  models  for  human 
limitations  that  will  be  described  below. 

MODEL  DESCRIPTION 

The  objective  of  this  section  of  the  paper  is  to  familiarize 
the  reader  with  the  basic  structure  of  the  OCM  and  to  provide  an 
appreciation  of  the  types  of  assumptions  and  considerations 
required  to  apply  the  model.  Discussion  is  largely  verbal  and 
conceptual;  the  reader  is  referred  to  the  literature  for 
mathematical  details  [1,2]. 

In  order  to  apply  the  OCM,  the  following  features  of  the 
environment  must  be  specified:  1)  a  linearized  state  variable 
representation  or  model  of  the  system  being  controlled;  2)  a 
stochastic  or  deterministic  representation  of  the  driving  function 
or  environmental  disturbances  over  which  the  operator  must  exert 
control;  3)  a  linearized  "display  vector"  summarizing  the  sensory 
information  utilized  by  the  operator  (including  visual,  vestibular 
and  other  sources  as  appropriate)  ;  and  4)  a  quantitative  statement 
of  the  criterion  or  performance  index  for  assessing 
operator /machine  performance.  The  specific  assumptions  concerning 
this  description  that  are  necessary  to  apply  the  theory  are  given 
in  reference  1. 

The  OCM,  diagrammed  in  Figure  1,  is  a  model  for  the  dynamical 
response  behavior  of  the  closed-loop  control  system.  Because  the 
model  is  capable  of  treating  multi-variable  systems,  all  system 
variables  shown  in  the(  figure  are  represented  as  vector  quantities. 
The  portion  of  the  model  structure  designated  as  "Human  Operator 
Model"  contains  elements  related  to  the  operator's  adaptive 
response  behavior  and, to  limitations  that  constrain  this  behavior. 
We  briefly  review  these  model  elements  in  the  order  corresponding 
to  the  flow  of  information  indicated  in  the  figure. 

First,  the  displayed  variables  are  assumed  to  be  corrupted  by 
"observation  noise"  introduced  by  the  human  operator.  This  noise 
is  analogous  to  the  internal  noise  level  postulated  in  signal 
detection  theory  and,  as  discussed  later,  provides  one  means  by 
which  the  model  accounts  for  human  limitations  in  perceptual 
resolution,  central-processing,  and  attention-sharing  capacity. 

At  this  point  the  model  is  dealing  with  a  noisy  representation 
of  the  displayed  quantities.  That  representation  is  then  delayed 
by  an  amount, t  ,  representing  internal  human  processing  delays. 
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DISTURBANCES  M(tl 


HUMAN  OPERATOR  MODEL 


Figure  1.  Structure  of  the  Optimal  Control  Model 


In  theory,  it  should  be  possible  to  consider  differential  delays 
for  different  sensory  channels,  but  this  additional  complication 
has  generally  been  unnecessary  in  past  model  applications  to  manual 
control  data. 

The  central  elements  of  the  model  are  represented  in  the 
blocks  described  as  the  Kalman  estimator  and  predictor.  Their 
purpose  is  to  generate  the  best  estimate  of  the  current  state  of 
system  variables,  based  on  the  noisy,  delayed  perceptual 
information  available.  These  elements  compute  the  estimate  of  this 
state  so  as  to  minimize  the  residual  estimation  uncertainty;  they 
represent  the  operator's  ability  to  construct  from  his 
understanding  of  the  system  and  his  incomplete  knowledge  of  the 
moment -by -moment  state  of  the  system,  a  set  of  expectancies 
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concerning  the  system  behavior  at  the  next  moment  in  time.  These 
elements  reflect  the  assumption  that  the  human  operator  has  both  an 
internal  model  of  the  dynamics  of  the  system  being  controlled,  and 
a  representation  of  the  statistics  of  the  disturbance  driving  the 
system. 

Given  the  best  estimate  of  the  current  system  state,  the  next 
model  element  (labeled  L*)  assigns  a  set  of  control  gains  or 
weighting  factors  to  the  elements  of  the  estimated  state,  in  order 
to  produce  control  actions  that  will  minimize  the  defined 
performance  criterion.  As  might  be  expected,  the  particular  choice 
of  the  performance  criterion  determines  the  weighting  factors,  and 
thus  the  effective  control  law  gains. 

Just  as  an  observation  noise  is  postulated  to  account  for 
perceptual  and  central  processing  inadequacies,  a  motor  noise  is 
introduced  to  account  for  an  inability  to  generate  noise-free 
control  actions.  In  many  applications  this  noise  level  is 
insignificant  in  comparison  to  the  observation  noise,  but  where 
very  precise  control  is  important  to  the  conditions  being  analyzed, 
motor  noise  can  assume  greater  significance  in  the  model.  Finally, 
the  noisy  control  response  is  assumed  to  be  smoothed  by  a  filter 
that  accounts  for  an  operator  bandwidth  constraint.  In  the  model, 
this  constraint  arises  directly  as  a  result  of  a  penalty  on 
excessive  control  rates  included  in  the  performance  criterion.  The 
constraint  may  mimic  actual  physiological  constraints  of  the 
neuromotor  system  or  it  may  reflect  subjective  limitations  imposed 
by  the  pilot. 

This,  then,  provides  a  conceptual  description  of  the  elements 
of  the  optimal  control  model.  It  should  be  emphasized  that  the 
parameter  values  that  must  be  provided  by  the  investigator 
correspond  to  the  human  limitations  that  constrain  behavior.  With 
these  limitations  as  the  constraints  within  which  performance  is 
produced,  the  model  predicts  the  best  that  the  operator  can  do.  A 
large  backlog  of  empirical  research  provides  the  data  necessary  to 
make  realistic  estimates  of  the  appropriate  parameter  settings  in 
the  manual  control  context.  This  research  has  shown  that,  for  an 
important  class  of  control  tasks,  these  parameters  are  relatively 
invariant  with  respect  to  changes  in  task  environment,  thus 
enhancing  the  model's  predictive  capability  [1-4], 

Let  us  now  review  in  more  detail  certain  critical  features  of 
the  model. 
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Task  Requirements 


The  pilot  is  assumed  to  adopt  a  control  strategy  that 
minimizes  a  quadratic  performance  index  of  the  following  form: 
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where  "q",  "r",  .d  "gB  are  the  cost  coefficients  associated  with 
display,  control,  and  control-rate  variables,  and  E  signifies 
statistical,  expectation.  Single-variable  laboratory  tracking  tasks 
can  often  be  modeled  under  the  assumption  that  the  operator 
minimizes  a  weighted  sum  of  mean-squared  error  and  mean-squared 
control  rate  [1-5] .  For  more  complex  tasks,  cost  coefficients  will 
be  associated  with  more  than  one  element  of  the  display  vector. 
When  analyzing  aircraft  approach-to-landing,  for  example,  penalties 
are  generally  associated  with  path,  attitude,  control,  displacement 
and  control  rate.  Reasonable  model  predictions  have  been  obtained 
on  the  basis  of  assumed  maximum  allowable  values  for  various 
control  and  response  variables  [6,7]  . 


The  performance  index  of  Eq(l)  leads  to  a  constant  set  of 
control  gains  because  of  the  assumption  that  “cost"  is  computed 
over  an  arbitrarily  long  time.  Although  most  model  applications 
have  been  based  on  this  assumption,  this  is  not  a  theoretical 
limitation  of  the  OCM,  and  tasks  (such  as  airplane  flare  and 
touchdown)  involving  "terminal  cost"  and  time- varying  control  gains 
can  be  treated. 


Perceptual.  Erv v  i  r  crime  n  t 


The  perce 

ptual  submodel  contained  in  the  ( 

1CM  is  represented  by 

the  following 

relationships  % 

y  (t)  =  Cx(t) 

(2) 

y  (t)  =  y  ( t)  +  v(t) 

(3) 

where  x(t)  is 

the  vector  of  system  "states". 

y(t)  is  the  display 

vector,  y  (t)  an  observation  noise  vector,  and  C  a  linear 
transformation  matrix. 

As  indicated  by  these  equations,  each  display  variable  to  the 
operator  is  assumed  to  be  a  linear  combination  of  system  states  and 
is  further  assumed  to  be  corrupted  by  a  white  noise  process.  Each 
noise  process  is  assumed  to  be  linearly  independent  of  other  such 
noise  processes  and  of  external  inputs  acting  on  the  system. 
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Because  the  OCM  is  an  informational  model  of  the  task 
environment  (including  the  human  operator) ,  the  perceptual 
environment  should  be  analyzed  to  determine  both  its  informational 
content  relevant  to  the  specific  task,  and  the  quality  of  such 
information  in  terms  of  allowing  the  operator  to  improve  his 
estimates  of  system  state  variables.  Informational  content  is 
reflected  in  the  composition  of  the  display  vector  This  vector 
should  include  each  potentially  relevant  cue  determined  from 
analysis  of  the  display  environment  and  described  as  a  linear 
combination  of  system  state  variables.  The  question  of  whether 
such  a  cue  is  of  sufficient  quality  to  be  useful  in  a  particular 
control  situation  is  a  separate  consideration  as  discussed  below. 


Identification  of  relevant  perceptual  cues  is  straightforward 
in  some  cases,  subtle  in  others.  Where  cues  are  obtained  only  from 
devices  that  present  symbolic  display  information  (e.g.,  aircraft 
flight-control  instruments) ,  one  generally  associates  two 
components  of  the  vector  with  each  moving  display  element:  one 
for  displacement,  and  one  for  velocity.  (Independent  perceptions 
of  position  and  velocity  are  assumed) .  On  the  other  hand,  the  set 
of  relevant  perceptual  cues  contained  in  an  actual  or  simulated 


"real-world"  visual  scene  may  not  be  apparent? 
scene  analysis,  such  as  geometric  analysis, 


some  form  of  visual 
may  be  required  to 


construct  a  suitable  display  vector  for  the  OCM  [8-10] . 


The  model  has  two  different  mechanisms  for  treating  the 
quality  of  perceptual  information:  (a)  adjustment  of  the 
statistical  properties  of  the  observation  noise,  and  (b) 
modification  of  the  description  of  system  dynamical  response 
characteristics  to  include  bandwidth  limitations  associated  with 
generation  and  perception  of  displayed  variables. 


For  an  idealized  perceptual  environment,  in  which  physical 
limitations  such  as  threshold-  and  saturation-like  phenomena  are 
negligible,  the  covariance  of  each  observation  noise  process  may  be 
considered  to  vary  in  proportion  to  the  variance  of  the  associated 
perceptual  variable.  A  "noise/signal  ratio"  of  around  -20  dB  is 
typical  of  a  wide  class  of  laboratory  tracking  tasks  [1]  .  If 
visual  or  indifference  thresholds  are  important,  such  as  in  many 
real  or  simulated  flight  tasks  (especially  those  involving  external 
visual  cues),  a  more  complex  formula  is  used  for  obtaining 
observation  noise  levels  [11,12] .  The  method  employed  involves  a 
statistical  threshold  that  results  in  large  observation  noise  for 
signal  magnitudes  below  the  assumed  threshold  value.  This  is 


directly  analogous  to  the  threshold  notions  of  signal  detection 
theory. 
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Effective  perceptual  thresholds  of  0.05  degrees  visual  arc  and 
0.2  arc  degrees/second  may  be  assumed  for  perception  of  indicator 
displacement  and  velocity,  respectively,  when  the  display  consists 
of  an  indicator  bar  that  translates  with  respect  to  a  stationary 
reference  marking  [3] .  The  literature  provides  some  guidance  for 
thresholds  appropriate  to  visual  scene  perception  as  well  as 
whole-body  motion  cues  [8,9,13,14];  but,  for  perceptual  cues  not 
heretofore  modeled,  some  experimentation  will  generally  be 
required  to  determine  appropriate  values  for  thresholds  related  to 
perceptual  resolution.  Indifference  thresholds,  which  reflect 
performance  requirements  rather  than  perceptual  resolution,  will 
depend  on  the  context  of  the  specific  tasks. 

Bandwidth-related  perceptual  limitations  may  result  from  the 
properties  of  the  physical  display  (e.g.,  inertia  and  damping 
associated  with  an  aircraft  attitude  indicator)  or  from  limitations 
associated  with  the  human  operator's  processing  of  certain  kinds  of 
physical  stimuli  (e.g.,  dynamical  response  properties  associated 
with  perception  of  whole-body  motion  cues  by  the  vestibular  system 
[13,14]).  Whatever  their  source,  perceptual  bandwidth  limitations 
are  expressed  in  linearized  state-variable  format  and  are  included 
in  the  description  of  system  response  dynamics. 

In  summary,  the  following  steps  are  required  to  define  the  OCM 
perceptual  submodel  in  a  specific  application: 

1.  Analyze  the  perceptual  environment  to  determine  the  relevant 
perceptual  cues  that  may  be  available  to  the  controller.  These 
cues  must  be  expressed  as  linear  combinations  of  system  state 
variables. 

2.  Modify  the  dynamical  equations  of  motion  of  the  system  to 
include  dynamical  response  limitations  associated  with 
perception  of  these  cues. 

3.  Determine  effective  perceptual  or  indifference  thresholds 
associated  with  each  perceptual  element,  and  include  those 
threshold  values  in  the  expressions  for  observation  noise 
variance. 

Control  Environment 

The  OCM  has  a  simplified  "built-in"  representation  of 
control-related  limitations.  First,  the  human  controller  is 
modeled  as  though  there  is  a  cost  penalty  associated  with  the 
rate-of-change  of  control  effort.  One  mathematical  consequence  of 
this  assumption  is  the  imposition  of  a  first-order  lag  network  — 
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characterized  by  the  "motor  time  constant"  T  —  associated  with 
the  operator's  response  strategy.  Second,  a  Gaussian  white  noise 
process  is  associated  with  rate-of-change  of  each  control  variable. 

For  single-variable  laboratory  tracking  tasks  involving 
fast-responding  systems,  motor  time  constants  on  the  order,  of  0.1 
seconds  have  provided  good  model  correspondence  with  data  [1-5] . 
The  relative  invariance  of  T  suggests  that  this  parameter  reflects 
an  inherent  bandwidth  limitation  of  human  controller  response.  For 
more  complex  and  realistic  tasks,  however,  response  bandwidth  is 
more  likely  to  be  limited  by  physical  constraints  associated  with 
the  control  system  [6,7]. 

As  currently  formulated,  the  OCM  allows  independent  adjustment 
of  two  motor  noise  processes:  a  "driving"  noise  to  reflect  direct 
inputs  to  the  control  system,  and  an  "internal"  noise  to  degrade 
the  model's  ability  to  estimate  certain  state  variables.  The 
driving  noise  potentially  allows  the  OCM  to  account  for  tremor  and 
other  inadvertent  disturbances  associated  with  execution  of 
control,  whereas  the  internal  noise  may  be  interpreted  as  a 
mathematical  device  to  account  partially  for  the  operator's 
imperfect  knowledge  of  system  response  behavior.  A  more  thorough 
discussion  of  the  treatment  of  motor  noise  may  be  found  in  Levison, 
Baron,  and  Junker  [15]. 

Typical  application  of  the  OCM  by  the  authors  involves 
omitting  the  driving  motor  noise  and  setting  the  covariance  of  the 
internal  motor  noise  to  be  on  the  order  of  -50  dB  with  respect  to 
control-rate  variance.  For  situations  in  which  control  resolution 
and  operator  tremor  become  important,  one  should  consider  a  motor 
noise  submodel  parallel  to  that  adopted  for  observation  noise; 
e.g.,  where  the  noise  increases  (or  at  least  levels  off)  as  the 
intended  control  action  becomes  small  compared  to  some  "control 
threshold".  Elaboration  of  this  aspect  of  the  OCM  is  an  area  for 
future  research. 

The  OCM  has  been  validated  with  data  obtained  largely  from 
tasks  involving  nearly-isometr ic  (i.e.,  force)  control  sticks.  In 
such  cases,  the  first-order  filter  represented  by  the  motor  time 
constant  T  has  been  sufficient  to  reflect  bandwidth  limitations 
that  may  have  been  imposed  by  the  operator's  neuromuscular  response 
mechanisms.  Where  substantial  control  movement  is  required, 
however,  higher-order  representations  are  needed  to  account  for  the 
combined  dynamical  characteristics  of  the  physical  control  device 
and  the  operator's  neuromuscular  system  [16]. 
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To  treat  second  or  higher-order  representations  of  operator 
response  limitations,  the  corresponding  dynamical  equations  of 
motion  are  included  in  an  expanded  formulation  of  system  response 
characteristics  (the  same  treatment  as  accorded  display-related 
dynamics) .  Some  studies  have  been  undertaken  to  develop  detailed 
dynamical  models  for  man-control  interaction  [17]  and  this  remains 
an  area  for  further  research. 

Attention-Sharing  and  Workload 

The  OCM  is  able  to  reproduce  pilot  response  behavior  in  a 
variety  of  idealized  laboratory  tracking  situations  with  an 
observation  noise/signal  ratio  that  is  nearly  invariant  across 
tasks  and  across  subject  populations.  The  near  constancy  of  this 
parameter  suggests  that  the  noise/signal  ratio  reflects  a  central 
processing  (rather  than  perceptual  or  motor)  limitation,  and  these 
results  have  led  to  the  following  model  for  central 
attention-sharing : 


where  ffc  is  the  fraction  of  attention  devoted  to  the  tracking  task 
as  a  wnole,  f.  is  the  subfraction  of  such  attention  devoted  to 
display  variable  y^ ,  P.  is  the  noise/signal  ratio  associated  with 
perception  of  y^ ,  ana  PQ  is  the  baseline  noise/signal  ratio 
associated  with  a  high-workload,  single-variable  tracking  task 
(typically,  -20  dB) .  According  to  this  submodel,  low  observation 
noise  is  associated  with  high  attentional  workload. 

The  general  nature  of  this  article  precludes  a  detailed  review 
of  the  theoretical  basis  for  relating  attentional  workload  to  the 
observation  noise/signal  parameter  of  the  pilot  model. 
Mathematical  development  is  provided  in  Ref  [18]  .  For  now,  let  us 
present  a  number  of  plausibility  arguments  to  support  this  notion: 

1.  The  more  the  pilot  suppresses  his  remnant  (i.e.,  noise 
component  of  his  control  response) ,  the  higher  his  workload. 

2.  Suppression  of  random  fluctuations  in  the  pilot's  response 
strategy  requires  mental  effort.  As  shown  by  Levison  and 
Kleinman  [19]  ,  rapid  fluctuations  in  the  response  strategy  may 
be  an  important  source  of  pilot  remnant. 

3.  Task  workload  is  directly  related  to  the  fraction  of  time  spent 
performing  the  task.  As  shown  by  Levison  [18],  an  inverse 
relationship  exists  between  the  fraction  of  time  devoted  to 
obtaining  information  from  a  display  variable  and  the 
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observation  noise  associated  with  that  display  when  scanning  is 
rapid  with  respect  to  the  signal  bandwidth. 

4.  For  information  sources  that  are  processed  in  parallel,  the 
workload  associated  with  a  particular  source  is  proportional  to 
the  information-processing  "capacity"  allocated  to  that  source. 

In  summary,  a  number  of  information-processing  attributes  that  we 
might  reasonably  associate  with  task  workload  can  be  related  to 
pilot  remnant  and,  therefore,  through  the  structure  of  the  OCM,  to 
the  noise/signal  parameter  given  in  Equation  (4) . 

The  workload  submodel  suggested  here  has  been  partially 
validated  in  studies  of  multi-variable  tracking  by  Levison,  Elkind, 
and  Ward  [4]  .  Wewerinke  [20]  has  also  obtained  generally  good 
agreement  between  subjective  workload  assessments  and  a  "workload 
index"  based  partly  on  this  submodel.  Some  of  this  validating  data 
is  presented  in  the  following  section. 

The  parameter  ffc,  representing  attention  to  the  task  as  a 
whole,  may  serve  as  a  metric  for  task  workload  [6,7];  f^  on  the 
other  hand,  reflects  penalties  associated  with  sharing  attention 
among  multiple  displays  [4] .  Attention-sharing  may  occur  with  a 
single  modality  or  across  modalities.  The  ^  may  be  based  on 
available  operator  scanning  data,  or  the  OCM  may1  be  used  to  predict 
optimal  allocation  of  attention  (i.e.,  the  attentional  allocation 
that  minimizes  the  overall  performance  index.) 

This  attention-sharing  model  is  crucial  for  predicting 
performance  in  complex,  multivariable  tasks,  and  it  can  also  serve 
as  a  basis  for  developing  a  variety  of  models  for  monitoring  tasks 
not  requiring  continuous  control  [21] .  In  the  applications  section 
of  this  paper  we  demonstrate  use  of  the  OCM  to  predict  the  effects 
of  various  system  parameters  on  the  relationship  between 
closed-loop  performance  and  attentional  workload. 
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VALIDATION 


Basic  Results 

Two  levels  of  model  validity  are  explored:  structural 
adequacy*  and  predictive  validity.  The  former  requirement  is 
satisfied  if  the  model  structure  and  parameterization  are 
sufficient  to  mimic  experimental  measurements  obtained  over  a 
useful  range  of  control  tasks.  A  model  that  is  structurally 
adequate  can  be  used  to  characterize  operator  response  behavior  in 
terms  of  a  few  parameters  (i.e.,  the  independent  model  parameters) 
and  therefore  has  the  potential  for  considerable  data  compression. 
Such  a  model  might  be  used,  say,  to  quantify  the  effects  of  stress 
and  other  environmental  factors  on  pilot  response  capabilities. 

The  more  stringent  property  of  predictive  validity  requires 
not  only  that  the  model  be  structurally  adequate,  but  that  operator 
and  closed-loop  system  response  behavior  be  predictable  with  a 
fixed  set  of  independent  model  parameters  (or  a  fixed  set  of  rules 
for  adjusting  such  parameters) .  A  model  of  this  sort  can  be  used 
for  system  design  and  evaluation  as  well  as  for  characterizing 
operator  performance,  as  described  later. 

To  demonstrate  structural  adequacy,  independent  model 
parameters  were  adjusted  to  provide  the  best  match  to  data  obtained 
from  three  different  laboratory  tracking  studies  which  utilized 
simulated  vehicle  dynamics  of  approximate  proportional  control  (Bp  , 
rate  control  (K/s) ,  and  approximate  acceleration  control  (K/s  ) . 
These  studies  are  detailed  in  References  [23,3,  and  24], 
respectively.  The  numerical  search  scheme  described  by  Levison 
[22,25]  was  employed  to  adjust  the  "motor  time  constant"  T  ,  the 
time  delay the  observation  noise/signal  ratios  associated  with 
perception  of  tracking  error  and  error  rate  (P  and  P^, 
respectively) ,  and  the  motor  noise/signal  ratio  (P  ) .  parameters 
providing  the  best  joint  match  to  variance  scores  and  frequency 
response  measures  are  given  in  the  first  three  rows  of  Table  1. 

Figure  2a  shows  that  the  linear  portion  of  the  pilot's 
response  strategy  ("gain"  and  "phase")  as  well  as  the  stochastic 


*  We  shy  away  from  the  term  "structural  validity" ,  which  would 
imply  a  close  correspondence  between  the  detailed  mathematical 
structure  of  the  model  and  the  human's  information  processing 
mechanisms.  Rather,  we  attempt  to  show  here  that  the  model 
structure  is  adequate  to  mimic  input-output  relationships 
exhibited  by  the  data. 
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Table  1:  Independent  Model  Parameters 


Dynamics 

pe 

(dB) 

P* 

e 

(dB) 

P 

u 

(dB) 

T 

(sec) 

Tn 

(sec) 

K 

-21.1 

-19.5 

-40.9 

.167 

.0812 

R/s2 

-23.6 

-18.1 

-41.6 

.154 

.0734 

K/s^ 

-  4.6 

-21.0 

-63.9 

.206 

.109 

(AVG) 

“ 

21.0 

-50.0 

0.17 

.09 

portion  of  the  control  response  ("remnant")  were  matched  quite 
closely.  In  addition,  RMS  error  and  control  scores  were  all 
matched  to  within  10%  of  experimental  values.  The  OCM,  then, 
appears  to  have  structural  adequacy  for  the  range  of  tasks 
explored. 

To  explore  the  predictive  validity  of  the  model,  a  single  set 
of  independent  model  parameters  (shown  in  the  last  row  of  Table  1) 
were  found  to  provide  the  best  joint  match  to  the  entire  body  of 
data  shown  in  Figure  2a.  Parameterization  was  reduced  by  one 
degree  of  freedom  by  constraining  Pe  and  P.g  to  have  identical 

numerical  values. 
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a)  Experimental  Conditions  b)  Best  Joint  Match  to 

Matched  Individually  Three  Conditions 


Figure  2.  Comparison  of  Model  and  Experimental  Frequency  Response 

Experimental  results  indicated  by  symbols,  model 
results  by  smooth  curves. 


FREQUENCY  (ftod/sec) 


Figure  3.  Effect  of  Attention  Sharing  on  Frequency  Response 
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Figure  2b  shows  good  correspondence  between  model  and 
experimental  results.  RMs  error  and  control  scores  (not  shown) 
were  also  generally  well  matched.  Now,  since  the  data  were  matched 
after  the  fact,  we  cannot,  strictly  speaking,  consider  these 
results  as  a  demonstration  of  the  model's  predictive  capability. 
Nevertheless,  the  ability  to  account  quite  well  for  the  (rather 
considerable)  changes  in  pilot  describing  functions  and  remnant 
spectra,  with  a  fixed  set  of  four  independent  model  parameters, 
suggests  that  the  model  can  be  used  a  predictor  of  performance 
trends,  at  the  very  least.  As  described  later,  this  capability  of 
the  model  has  been  helpful  in  the  design  of  simulator  experiments. 


As  one  might  expect,  the  ability  of  the  model  to  generate 
accurate  predictions  of  pilot/vehicle  performance,  using  the  four 
parameter  values  shown  at  the  bottom  of  Table  1,  degrades  as  the 
complexity  of  the  task  increases.  A  recent  study  suggests  that 
both  the  motor  time  constant  and,  to  a  lesser  extent,  observation 
noise  ratio,  increase  with  increasing  order  and/or  delay  in  the 
plant  dynamics  [22] .  This  finding  suggests  that  the  current 
formulation  of  independent  model  parameters  does  not  completely 
isolate  pilot  response  capabilities  from  task  parameters.  Work  is 
currently  in  progress  to  effect  this  isolation  and  thereby  improve 
the  absolute  predictive  capability  of  the  model. 

Attention-Sharing 


A  set  of  experiments  was  performed  by  Levison,  Elkind,  and 
Ward  [4]  to  test  the  submodel  for  workload  and  attention-sharing 
described  above.  Subjects  were  provided’ with  two  2-axis  controls 
and  four  separated  displays  and  were  required  to  perform  up  to  four 
concurrent  linearly  independent  rate-control  tracking  tasks.  The 
tasks  were  performed  singly  and  in  combination.  When  performing 
multiple  tasks,  fixation  was  maintained  on  a  single  display  and  the 
remaining  three  displays  were  tracked  using  peripheral  vision 
(i.e.,  scanning  was  not  allowed). 


The  single-variable  tasks  were  used  as  "calibration" 
experiments  to  define  numerical  values  for  pilot-related  parameters 
(including  a  "residual  noise"  term  that  was  added  to  the  expression 
for  observation  noise  to  account  for  peripheral  viewing) .  With  all 
model  parameters  held  constant  (except  for  noise/signal  ratio) , 
both  attentional  allocation  and  system  performance  were  predicted 
for  the  4-axis  tracking  task.  Predictions  were  based  on  the 
assumptions  that  total  task  workload,  ffc,  remains  constant  for  the 
single-  and  multi-axis  tasks,  and  that  the  f.  sum  to  unity. 
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; 

There  was  considerable  interference  among  tasks;  that  is, 
subtask  and  total-task  scores  were  greater  when  the  tasks  were 
performed  concurrently  than  when  performed  singly.  Model 
predictions  for  the  multiple-task  case  were  in  good  agreement  with 
the  data,  especially  for  the  total-task  score  (which  is  the 
quantity  the  subjects  were  instructed  to  minimize) .  Agreement  with 
some  of  the  subtask  scores  was  less  good,  apparently  because 
total-task  performance  was  relatively  insensitive  to  allocation  of 
attention  among  the  component  tasks. 

Figure  3  shows  the  effect  of  attention-sharing  on  the 
normalized  observation  noise  (a  linear  transformation  of  the 
pilot's  remnant  spectrum)  and  on  the  pilot's  describing  function 
(i.e.,  control  strategy).  Results  are  presented  for  the 

foveally-f ixated  display.  Model  results  agree  quite  well  with  the 
data  and  show  that  the  effects  of  multiple-task  requirements  were 
to  increase  the  observation  noise  spectrum  (as  predicted  by  the 
model  for  attention-sharing)  and  to  decrease  the  amplitude  ratio 
and  increase  high-frequency  phase  lag  of  the  describing  function 
(an  adaptive  response  by  the  pilot  to  filter  out  some  of  the 
effects  of  increased  observation  noise. 

The  model  relating  attention  to  performance  has  also  been 
validated  for  decision  tasks  and,  to  some  extent,  for  combined 
decision  and  control  tasks.  Levison  and  Tanner  [26]  showed  that 
the  model  for  attention-sharing  provided  excellent  agreement  with 
the  observed  decrement  in  performance  between  a  single  decision 
task  and  two  concurrent  tasks.  Predictions  were  less  accurate  for 
concurrent  decision  and  control  tasks;  combined-task  performance 
suggested  that  some  interference  was  present,  but  to  a  lesser 
degree  than  predicted  by  the  constraint  of  fixed  total-task 
workload. 


Multi-Sensory  Cue  Environments 

The  OCM  has  also  been  used  to  model  continuous  control 
performance  in  a  multi-sensory  cue  environment.  Levison  and  Junker 
[27,28]  studied  roll-axis  tracking  in  disturbance-regulation  and 
target-following  tasks.  They  compared  performance  when  only  visual 
cues  were  available,  to  performance  when  the  visual  cues  were 
augmented  with  platform  motion  cues.  As  shown  in  Figure  4,  they 


565 


Bolt  Beranek 


and  Newman 


Inc. 


found  that  the  OCM  could  provide  a  task-independent  framework  for 
explaining  performance  under  all  experimental  conditions.  The 
availability  of  motion  cues  was  modeled  by  augmenting  the  set  of 
perceptual  variables  to  include  position,  rate,  acceleration,  and 
acceleration-rate  of  the  motion  simulator.  This  straightforward 
informational  treatment  allowed  the  model  to  replicate  the  effects 
of  motion  cues  on  a  variety  of  response  measures,  for  both  the 
target-following  and  disturbance-regulation  tasks.  Model  results 
were  obtained  with  a  fixed  set  of  independent  model  parameters. 


(a)  TARGET  INPUT 


<D)  DISTURBANCE  INPUT 


Figure  4.  Comparison  of  Model  and  Experimental  Frequency  Response 

"Static":  visual  cues  only.  "Motion":  combined  visual 

and  motion  cues. 


APPLICATIONS 

The  OCM  has  been  applied  —  mostly  with  regard  to  aircraft 
flight  —  as  a  predictive  and  as  a  diagnostic  tool.  Areas  of 
application  include  display  design  and  evaluation,  control  design 
and  evaluation,  prediction  of  aircraft  handling  qualities, 
simulator  design  and  evaluation,  effects  of  environmental  stress, 
and  design  of  experiments.  Examples  in  each  area  are  cited  below. 
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Display  Design  and  Evaluation 

The  OCM  has  been  used  in  numerous  studies  to  evaluate  the 
effectiveness  of  aircraft  flight  displays.  In  some  cases,  model 
predictions  have  been  compared  with  data;  in  others,  the  OCM  has 
been  employed  as  a  purely  predictive  tool. 

Kleinman  and  Baron  [29]  analyzed  a  piloted  approach-to-landing 
task  to  evaluate  pictorial  display  requirements.  This  problem 
involved  a  time-varying  information  base  for  the  pilot.  The 
effects  of  different  display  formats  and  display  symbology  were 
predicted  in  cases  where  the  aircraft  was  subjected  to  turbulence 
and/or  constant  updrafts.  The  ability  of  the  pilot  to  estimate 
these  external  disturbances,  and  take  the  appropriate  corrective 

action  to  minimize  glide  path  errors  was  analyzed.  Predictions  of 
system  performance  were  compared  with  data  obtained  in  independent 
experimental  investigations.  The  model-data  agreements  were 
excellent  and  demonstrated  the  model's  ability  to  predict  the 
time-varying  adaptability  of  a  pilot  to  updraft  disturbances.  In 
addition,  the  agreement  between  model  results  and  data  for  cases  in 
which  there  was  no  turbulence  disturbing  the  aircraft,  provided 
further  evidence  of  the  validity  of  the  model  for  human  randomness 
(remnant) . 


Figure  5:  Effect  of  Lateral  Director  on  Predicted  Missed 
Approach  Probability 
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Figure  5  illustrates  a  typical  model  application  in  a  study  by 

Baron  and  Levison  [7]  to  develop  an  OCM-based  display  methodology 
for  Short  Takeoff  and  Landing  aircraft.  Two  display  configurations 
were  considered:  (1)  "status",  or  standard  symbolic  display,  and 
(2)  a  "flight  director"  display  that  combined  bank  angle,  sideslip, 
and  lateral  path  error  in  a  single  command  display.  Two  conditions 
of  atmospheric  turbulence  were  also  considered. 

A  quadratic  cost  criterion  based  on  landing-approach 
requirements  was  formulated,  and  model  predictions  were  obtained 
for  each  configuration  as  a  function  of  observation  noise/signal 
ratio.  To  generate  the  curves  shown  in  the  figure,  noise/signal 
ratio  was  interpreted  as  relative  attention  per  the  submodel  of 
Eq(4),  and  the  quadratic  performance  index  was  transformed  into  a 
probability  of  missed  approach  (i.e.,  probability  that  certain 
system  variables  would  be  outside  desired  limits) . 

Figure  5  shows  that  the  model  predicted  reasonable  performance 
and  workload  trends;  i.e.,  performance  was  expected  to  be  superior 
with  the  director  for  a  given  level  of  attention  (workload) ,  and 
workload  was  expected  to  be  less  with  the  director  for  a  given 
level  of  performance.  (This  study  was  analytic  in  nature  and  did 
not  involve  experimental  verification) . 

A  similar  approach  was  followed  to  analyze  both  display  and 
control  characteristics  for  an  aircraft  with  an  advanced  avionics 
configuration  [6]  ;  in  this  study,  performance  in  a  simulated 
windshear  environment  was  explored  both  analytically  and 
experimentally.  Hess  and  Wheat  used  the  OCM  to  analyze  an 
electronic  display  for  a  helicopter  [30] . 

The  OCM  has  also  been  applied  to  the  design  of  aircraft  flight 
displays.  Levison  [31]  suggested  a  procedure  for  designing  flight 
directors  in  which  the  director  laws  would  incorporate  optimal 
processing  of  the  display  parameters  as  predicted  by  the  OCM.  Hess 
[32]  has  proposed  a  similar  director  design  scheme  which  he  has 
incorporated  into  a  more  formal  display  design  procedure. 

Control  Design  and  Evaluation 

Although  display  problems  have  received  the  most  attention, 
other  aspects  of  the  system  design  problem  have  not  been  neglected 
completely.  Levison  and  Houck  [16]  have  explored  the  use  of  the 
model  in  analyzing  control  stick  design  problems  in  a  vibration 
environment.  Stengel  and  Broussard  [33]  have  used  the  basic 
structure  of  the  OCM  along  with  some  assumptions  concerning 
sub-optimal  adaptation  to  determine  stability  boundaries  in  high-g 
maneuvering  flight.  And,  recently,  Schmidt  [34]  has  proposed  a 
design  procedure  for  stability  augmentation  systems  based  on 
closed-loop  analysis  with  the  OCM. 
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The  study  of  Stengel  and  Broussard  is  particularly  interesting 
because  it  illustrates  an  application  of  the  model  involving 
imperfect  knowledge  by  the  pilot  of  his  vehicle  response  dynamics. 
The  flight  task  explored  in  this  study  required  the  pilot  to 
transition  from  a  low  angle-of-attack  (AOA)  flight  phase  to  one 
involving  high  AOA.  Manned  simulation  results  were  compatible  with 
the  notion  that  the  test  pilots  retained  internal  models 
appropriate  to  low  AOA  flight  (in  which  they  had  been  trained)  when 
in  the  high  AOA  phase. 

Prediction  of  Pilot  Opinion  Ratings 

Display  and  control  design  are  but  two  of  the  factors  that 
influence  a  vehicle's  "handling  qualities"  as  reflected  in  the 
pilot's  ability  to  achieve  acceptable  system  performance  at 
reasonable  levels  of  mental  workload.  Although  aircraft  handling 
qualities  are  specified,  for  the  most  part,  in  terms  of  vehicle 
response  characteristics  alone,  the  formal  acquisition  of 
subjective  pilot  opinion  is  an  important  aspect  of  aircraft 
elevation  and  acceptance.  Thus,  a  need  exists  for  a  reliable 
analytic  tool  for  predicting  pilot  opinion  ratings,  especially  for 
new  aircraft  configurations  and  task  environments. 

Hess  [35]  noticed  that,  for  a  variety  of  experimental  results 
that  he  matched  with  the  OCM,  the  objective  performance  index 
varied  monotonically  with  subjective  pilot  opinion  ratings,  and  he 
suggested  use  of  the  OCM  as  a  predictor  for  pilot  ratings. 
Building  on  this  effort,  and  on  earlier  work  by  Anderson  [36]  using 
servo-theory  pilot  models,  Levison  [37]  suggested  a  scheme  whereby 
model  predictions  of  system  performance  and  pilot  workload  could  be 
combined  to  predict  pilot  opinion  ratings. 

Figure  6  illustrates  application  of  the  OCM-based  prediction 
scheme.  Performance/workload  predictions  are  shown  for  eight  of 
the  vehicle  configurations  explored  by  Rickard  [38]  in  a  manned 
simulation  study  of  longitudinal  handling  qualities  for  large 
transport  aircraft  in  approach  to  landing.  Model  predictions 
generally  mimicked  the  trends  exhibited  by  subjective  opinion 
ratings:  the  more  favorably  rated  configurations  tend  to  lie  in 
the  lower  left-hand  corner  of  the  figure  (corresponding  to  good 
performance  and  low  workload) ,  whereas  the  less  favorably  rated 
conditions  JLie  toward  the  upper  right.  A  set  of  "rating 
expressions"  was  derived  to  obtain,  from  the  model  results  shown  in 
Figure  6,  "predictions"  of  opinion  ratings  that  corresponded  well 
with  experimentally-obtained  ratings. 
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RELATIVE  ATTENTION 


Figure  6.  Prediction  of  Performance  Vs.  Attentional  Workload 
for  Eight  Aircraft  Configurations 


A  subsequent  study  was  performed  to  provide  a  further  test  of 
the  model-based  handling  qualities  assessment  scheme  [39].  In  this 
study,  model  predictions  were  obtained  prior  to  a  manned  simulation 
study  in  which  both  objective  as  well  as  subjective  performance 
measures  were  obtained.  Although  model  predictions  proved  to  be 
optimistic,  compared  to  the  experimental  results,  performance 
trends  were  reliably  forecast.  Specifically,  the  analytic  scheme 
correctly  identified  the  vehicle  configurations  that  had  the  most 
adverse  pilot  opinion  ratings  and  the  largest  rms  excursions  in 
aircraft  state  and  control  activity. 
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Simulator  Design  and  Evaluation 

The  increased  interest  in  flight  simulators  has  spurred  some 
additional  extensions  and  applications  of  the  model.  Grunwald  and 
Merhav  [8]  and  Wewerinke  [9,40]  have  incorporated  mechanisms  for 
describing  the  utilization  of  extra-cockpit  visual  scene  cues  in 
the  OCM  and  have  obtained  preliminary  experimental  validation  of 
their  approaches.  Although  the  subtleties  and  complexities 
associated  with  human  perception  of  a  complex  scene  are  by  no  means 
resolved,  these  studies  do  suggest  that  the  OCM  could  be  useful  for 
analyzing  closed-loop  control  behavior  based  on  visual  scene  cues. 

In  a  somewhat  different  vein,  Baron,  Muralidharan  and  Kleinman 
[41]  used  the  OCM  to  develop  a  closed-loop  model  for  analyzing 
engineering  requirements  for  flight  simulators.  They  predicted  the 
effects  on  performance  of  certain  simulation  design  parameters, 
such  as  integration  scheme,  sample  rate,  etc.  Model  predictions 
were  later  verified  in  an  empirical  study  by  Ashworth  [42] . 

Environmental  Stress 

Military  operational  environments  may  subject  the  pilot  or  gun 
operator  to  substantial  physical  stress.  In  some  cases,  the  stress 
is  a  direct  consequence  of  the  flight  task  (e.g.,  vibration, 
sustained  high  acceleration) ;  in  other  cases,  stress  may  be  induced 
by  an  opponent  as  a  defensive  measure  (e.g.,  optical 
countermeasure)  .  Such  considerations  have  motivated  application  of 
the  OCM  to  tasks  involving  actual  and  simulated  environmental 
stress. 

A  series  of  studies  was  conducted  to  develop  a  methodology  for 
modeling  the  effects  of  high-frequency  vibration  on  pilot  response 
behavior  and  total  system  performance  [16,43-45].  This  effort  led 
to  a  model  structure  which  combines  the  OCM  with  a  biodynamic  model 
of  the  human  operator  [46].  As  part  of  this  structure,  a  set  of 
rules  were  developed  for  relating  certain  OCM  parameters 
(specifically,  time  delay  and  motor  noise  covariance)  to  biodynamic 
response  parameters. 

Korn,  Ephrath,  and  Kleinman  [47]  have  recently  applied  the  OCM 
to  a  study  of  the  effects  of  sustained  positive  linear  acceleration 
(G  stress.  The  acceleration  stress  degraded  system  performance; 
these  effects  were  modeled  by  increases  in  all  pilot-related  OCM 
parameters.  In  a  subsequent  study,  Korn  and  Kleinman  [48]  modeled 
the  effects  of  lateral  acceleration  stress  (G  )  by  changes  in  OCM 
parameters  consistent  with  the  notion  that*  the  subjects  were 
willing  to  tolerate  larger  errors  when  subjected  to  acceleration 
stress. 


571 


Bolt  Beranek  and  Newman  Inc. 


Zacharias  and  Levison  [5]  have  explored  use  of  the  OCM  as  a 
diagnostic  tool  for  identifying  the  particular  type  of 
information-processing  decrements  caused  by  environmental  stress. 
They  showed  that  their  identification  procedure  would  be  adequate 
to  differentiate  between  an  increase  in  information-processing  time 
and  a  decrease  in  visual  resolution  capabilities. 

Design  of  Simulation  Experiments 

This  author  and  his  colleagues  have  routinely  used  the  OCM  in 
the  design  phase  of  simulation  experiments.  Typically,  the  model 
is  used  to  select  certain  experimental  parameters  to  maximize  the 
effect  of  the  principal  experimental  variables. 

A  design  application  of  this  type  has  been  documented  by 
Junker  and  Levison  [49]  ,  who  used  the  OCM  to  design  the  motion-cue 
study  described  above  in  the  discussion  of  model  validation.  Since 
one  of  the  goals  of  that  study  was  to  determine  how  to  model  the 
effects  of  motion  cues,  pre-experiment  model  runs  were  made  to 
determine  forcing-function  spectra  and  performance  objectives  so 
that  (1)  motion  would  significantly  improve  performance  in  one 
experimental  condition,  and  (2)  motion  cues  would  have  no 
significant  effect  in  another.  Although  pre-experiment  model 
results  did  not  match  experimental  results  as  closely  as  shown  in 
Figure  4,  the  trends  of  the  results  were  quite  well  predicted,  and 
a  useful  experiment  was  consequently  performed. 

The  OCM  was  used  in  a  subsequent  study  to  design  an  experiment 
exploring  the  pilot's  use  of  the  "tilt  cue"  (i.e.,  the  apparent 
rotation  of  the  gravity  vector  that  is  felt  when  a  ground-based 
simulator  undergoes  pure  roll-axis  displacement)  [50] . 
Considerable  pre-experiment  analysis  was  required  to  find  a  set  of 
vehicle  dynamics  such  that  the  tilt  cue  would  significantly 
influence  tracking  performance.  Again,  experimental  performance 
trends  were  as  predicted  by  the  model. 

FURTHER  DEVELOPMENT  AND  APPLICATION 

Studies  are  currently  in  progress  to  extend  the  capabilities 
of  the  optimal  control  model  and  to  apply  it  to  a  wider  variety  of 
tasks.  One  such  study  is  expected  to  indicate  how  the  predictive 
accuracy  of  the  OCM  may  be  improved  (Air  Force  Contract  No. 
F33615-81-C-0517)  .  As  noted  in  the  discussion  on  model  validation, 
reliable  performance  trends  can  often  be  obtained  using  a  fixed  set 
of  values  for  pilot-related  model  parameters.  Nevertheless,  if  we 
consider  the  full  range  of  laboratory  results,  certain  systematic 
deviations  in  these  parameters  are  observed. 


572 


Bolt  Beranek  and  Newman  Inc 


The  consistency  of  such  parameter  deviations  suggests,  first, 
that  parameters  identified  as  "pilot-related"  reflect,  in  part,  the 
interaction  between  the  human  operator  and  the  task  environment; 
and,  second,  that  these  deviations  should  be  predictable  with  a 
suitable  model  structure.  Therefore,  in  order  to  improve  the 
predictive  capabilities  of  the  OCM  —  especially  in  terms  of 
predicting  absolute  performance  --  attempts  will  be  made  to  isolate 
parameters  that  more  nearly  reflect  human  response  limitations. 

The  study  referenced  above  will  also  address  the  development 
of  models  for  "learning  behavior"  (i.e.,  control-strategy 
development)  —  one  of  the  important  remaining  theoretical 
frontiers  in  manual  control.  Models  of  this  soft  have  ready 
application  to  the  design  of  training  simulators:  e.g.,  predicting 
for  a  given  type  of  flight  task  whether  or  not  whole-body  motion 
cues  will  reduce  flight  training  time.  It  is  not  necessarily 
sufficient  to  predict  the  effect  of  motion  cues  on  the  performance 
of  a  fully  trained  pilot;  rather,  the  question  is  the  extent  to 
which  motion  cues  might  allow  the  trainee  more  rapidly  to  develop 
the  correct  response  strategy  (say,  by  more  rapidly  developing  a 
correct  "internal  model"  of  the  task  environment) . 

One  approach  being  considered  is  to  include  a  fourth  adaptive 
element  to  the  pilot  model:  an  optimal  "identifier"  to  account  for 
development  of  the  pilot's  internal  model  in  a  given  task 
situation.  This  model  element  would  account  both  for  the  rate  of 
learning  as  well  as  the  asymptotic  structure  of  the  internal  model 
(which,  in  complex  tasks,  may  be  different  from  the  true  system) . 

Because  of  high  lateral  accelerations  that  may  be  induced  by 
modern  military  aircraft  that  allow  direct  control  of  side  force, 
techniques  are  desired  to  optimize  the  design  of  control  sticks  in 
high-g  environments.  To  this  end,  a  study  is  in  progress  to  better 
define  the  pilot/stick  interface  (Air  Force  Aerospace  Medical 
Research  Laboratory  Project  No.  L72311708)  .  Three  control  devices, 
ranging  from  a  near-isometric  device  to  one  that  has  only  weakly 
restrained  displacement,  are  to  be  explored  in  a  fixed-base 
simulation  study.  Three  electrical  gains  will  be  explored  for  each 
stick.  The  results  of  this  study  are  expected  to  yield,  in  part,  a 
better  definition  of  the  motor  sub-model  of  the  OCM. 

A  study  is  also  in  progress  to  better  define  the  perceptual 
limitations  of  the  human  operator  (Air  Force  Contract  No. 
F33615-81-C-0515) .  This  analytical  and  experimental  study  will 
explore  the  pilots  of  non-visual  motion  cues  (platform  motion  and 
g-seat  cuing)  as  well  as  the  use  of  visual  scene  cues  relevant  to 
the  task  of  high-speed  contour-following.  Preliminary  work  in  the 
area  of  modeling  visual  scene  cues  has  been  reported  [10] . 
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The  major  opportunity  for  extending  the  OCM  to  other  tasks  is 
in  the  area  of  supervisory  control.  These  control  problems,  as  we 
have  noted,  involve  monitoring,  detection,  decision-making,  and 
discrete  and/or  infrequent  control.  Most  often,  the  systems  are 
highly  automated,  require  more  than  one  operator  and  are  extremely 
complex.  The  principal  feature  of  the  OCM  that  is  useful  for  these 
application  is  its  information  processing  structure,  but  the 
underlying,  normative  modelling  perspective  is  also  important.  The 
OCM  has  been  applied  to  tasks  of  this  sort  [21,26,51-54]  and 
additional  studies  are  contemplated  to  develop  and  apply  an 
OCM-based  model  to  the  study  of  flight  crew  performance. 

The  application  of  the  OCM,  or  its  derivatives  to  such 
problems,  is  in  its  early  stages  and  many  questions  remain  to  be 
answered.  Among  the  more  important  are  questions  concerning  the 
human's  internal  model  for  such  large  scale  systems,  the 
appropriate  control  and  decision  cost  functionals  for  these 
problems,  the  modeling  of  attention-sharing  strategies  in 
time-varying  situations,  and  the  appropriate  level  for 
incorporating  and  modeling  aspects  of  the  tasks  that  are  important 
but  are  not  likely  to  be  treated  using  the  same  techniques  (e.g., 
certain  types  of  intellectual  activity  or  procedural  tasks) . 
Finally,  there  is  a  need  for  additional  data  and,  indeed,  for 
appropriate  experimental  paradigms,  for  establishing  these  models. 
It  must  be  recognized  that  validation  of  such  complex  models  to  the 
degree  that  manual  control  models  have  been  and  can  be  validated, 
will  not  be  possible  for  some  time,  for  both  theoretical  and 
practical  reasons.  One  may  not  expect,  therefore,  that  supervisory 
control  models  that  are  "predictive",  in  the  same  sense  as  the  OCM, 
will  be  developed  in  the  near  future.  Nonetheless,  it  should  be 
possible  to  develop  models  of  supervisory  control  that  will  capture 
many  essential  features  of  tasks  of  interest  and  will  prove  to  be 
useful  design,  analysis,  and  evaluation  tools. 
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Introduction 


In  this  paper,  we  will  propose  a  method  for  the  identification  of  the 
pilot's  control  compensation  using  time  domain  techniques.  From  this  in¬ 
formation  we  hope  to  infer  a  quadratic  cost  function,  supported  by  the  data, 
that  represents  a  reasonable  expression  for  the  pilot's  control  objective 
in  the  task  being  performed,  or  an  inferred  piloting  "strategy".  (Note 
here  that  we  are  using  the  term  strategy  as  synonomous  with  control  objective, 
and  not  with  control  law.) 

The  ultimate  goals  of  this  research  topic  include  a  better  understanding 
of  the  fundamental  piloting  techniques  in  complex  tasks,  such  as  landing 
approach;  the  development  of  a  metric  measurable  in  simulations  and  flight 
test  that  correlate  with  subjective  pilot  opinion;  and  to  further  validate 
pilot  models  and  pilot-vehicle  analysis  methods.  At  this  time  we  will  present 
the  methodology  and  some  preliminary  numerical  results. 

The  Pilot  Model  and  Objective  Function 

The  analyses  relies  on  the  well-known  £lj  optimal -control  theoretic 
technique  for  modeling  the  human  pilot's  manual  control  function.  The 
hypothesis  upon  which  it  is  based  is  that  the  well  trained,  well  motivated 
pilot  chooses  his  control  inputs  (e.g.  stick  force)  to  meet  the  pilot's 
(internal)  objective  in  the  task,  subject  to  his  human  limitations.  His 
objective  is  further  assumed  to  be  expressible  in  terms  of  a  quadratic 
"cost"  function 


where  Yp  =  vector  of  pilot's  observed  variables  (e.g.,  attitude,  acceleration) 
Up  =  vector  of  pilot's  control  inputs 
Q,R,G  =  Pilot-Selected  (internal)  weightings 
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The  human  limitations  modeled  include  information-acquisition  and  processing 
time  delay,  observation  and  control  input  errors,  and  neuromuscular  dynamics. 
A  block  diagram  of  the  resulting  model  structure  is  shown  in  Figure  1. 

The  components  of  this  model  may  be  grouped  into  two  parts,  one  dealing 
with  the  information  acquisition  and  state  estimation,  and  one  related  to 
the  control  law  or  control  policy  operating  on  the  estimated  state.  As  has 
been  shown  in  the  references  on  this  modeling  approach,  the  "solution"  for 


the  pilot's  control  inputs,  as  predicted  by  the  model,  is  expressed  as 

.  j  *  T 
u  =  q  x  +  q  u  +  v 
p  yx  au  p  u 


where  x  =  internal  estimate  of  the  system  state 
gx.  9U  =  control  gains 
vu  =  motor  noise,  or  control  input  errors 

(Readers  unfamiliar  with  the  further  details  of  the  model  are  referred  to 
the  reference.) 

The  key  points  germain  to  this  analysis  are  that  the  above  equation  is 
a  mathematical  expression  representing  the  pilot's  overt  control  actions 
(Up),  and  these  control  actions  are  measurable  experimentally.  Furthermore, 
the  gains  gx  and  gu  are  functions  of  the  plant  (vehicle)  dynamics  and  his 
objective  function,  and  thereby  represent  his  control  "techniques",  level 
of  skill,  and  familiarity  with  the  vehicle  dynamics. 

Another  factor  of  importance  is  that  not  only  is  the  objective  function, 
from  which  the  gains  are  determined,  a  mathematical  part  of  a  pilot  control 
model,  but  it's  resulting  magnitude  obtained  from  exercising  the  model  has 
been  found  to  correlate  with  the  subjective  pilot  opinion  obtained  from 
simulation  and  flight  test.  Such  a  correlation  is  shown  in  Figure  2,  as  an 
example,  taken  from  Refs.  2  and  3.  This  of  course  assumes  one  has  been  able 
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to  correctly  express  the  pilot's  (internal)  cost  function,  which  is  in 
fact  his  strategy  that  depends  on  his  perception  of  the  task.  Now  this  is 
easy  to  do  in  simple  laboratory  tasks  in  which  the  subject  has  been  instructed 
to  minimize  some  displayed  error,  for  example.  But  it  is  not  at  all  clear 
just  what  flight  parameters  are  being  "regulated"  or  "tracked",  other  than 
ILS  glide  slope  and  localizer  error  in  the  case  of  landing  approach.  This 
is  but  one  example,  other  complex  piloting  tasks  might  be  considered  equally 
as  well . 


The  Identification  Procedure 

We  seek  then  a  method  by  which  we  may  identify  those  pilot  parameters 
that  reflect  his  control  techniques,  or  control  strategy.  Referring  back 
to  the  pilot  model  control  law,  or 


°p  *  *£  5  +  sj  up  + 


we  note  that  the  gains  gx  operate  on  the  estimated  state  x.  Now  the  separation 
principle  of  optimal  estimation  and  control  theory  states  that  the  control 
gains  (gx,  gu)  are  independent  of  the  state  estimation  process.  Further, 
the  optimal  state  estimator,  in  general  and  in  the  pilot  model,  is  independent 
of  the  overall  objective  function  being  minimized  by  the  controller  (estimator 
and  control)  law.  Therefore,  if  we  are  mainly  after  the  pilot's  control 
strategy  as  expressed  by,  or  at  least  a  function  of,  his  objective  function, 
we  need  only  to  focus  on  the  gains  (gx,  gu)  and  not  on  those  variables  related 
only  to  the  state  estimator.  These  latter  variables  include  the  time  delay, 
and  observation  and  motor  noise  covariance  matrices,  parameters  of  interest 
in  the  identification  technique  of  Levison  ,  for  example.  If  our  approach 
is  successful,  fewer  parameters  must  be  identified  from  the  data,  which  is 
always  an  advantage,  but  the  parameters  affecting  the  estimation  process  are 
assumed. 
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The  identification  method  proposed  is  as  follows.  The  control  law 
expressed  previously,  may  be  rewritten  as 

T  T  T 

u  =  q  X-q  e  +  q  u+v 
p  ax  ax  au  p  u 

where  e  =  error  in  estimating  the  true  (actual)  state  x.  Note  that  along 
with  the  pilot's  control  Up,  these  true  states,  such  as  angle  of  attack  or 
pitch  attitude  are  measureabl e,  but  the  state  estimate,  x,  is  a  quantity 

a 

internal  in  to  pilot,  as  modeled.  Hence  x  is  not  measurable  — nor  are 
e  or  vy.  Transposing  the  above,  multiplying  by  x  =  col  £x,  Upj,  and  taking 
expected  values  yields 


E(xxT)  'E(xuJ) 

E(xeT)  j  0 

*gx" 

s 

- 

1 

E(u  uT) 

L  p  p  J 

.e(VT)!«vJ). 

E(upcT>  j  0  _ 

4 

.V 

or  N; 


+  N 

v, 


’E(xvJ) 

.e<vT)- 

v  u 


(I) 


Now  to  evaluate  these  matrices  we  note  first  that,  in  a  simulation  at 

least,  the  vectors  x(t)  and  Up(t)  are  measurable,  so  estimates  of  their 

covariance  matrices  (e.g.,  E(xx^))may  be  obtained  from  measurements  of 

sampled  time  histories.  (Also,  in  this  paper  we  assume  that  good  estimates 

of  u  are  available  from  filtered  measurements  of  u  .  The  details  of 
P  p 

accomplishing  this  filtering  are  under  current  investigation,  but  digital 
techniques  as  well  as  analog  methods  are  still  available.)  For  reference, 
refer  to  Figure  3. 
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With  regard  to  the  remaining  terms  involving  e  and  vu,  both  are  not 
measurable  and  need  attention.  To  resolve  this  consider  the  complete 
system  dynamics  model  by  the  relation 


*  *  Ax  +  BUp  +  w 
and 

"p  ■  $  -  +  9J  up  + 

A 

where  the  relation  between  state  and  estimate,  or  x  =  x  -  e  has  been  employed. 
The  pilot's  internal  state  estimation  error,  e,  is  treated  as  follows.  Define 
eu  =  Up  -  Up  to  be  the  error  in  estimation  of  the  pilot's  own  control  input, 
and  then  let 


Now  the  covariance  of  e  may  bd  shown  to  be  governed  by  the  relation 

cov  (7)  =  E(7  7T)  ^  P 
P  *  Aj  P  +  Pa{  +  Wj 


Also  we  have 


and 


Aj  E  +  r  Aj  +  Wx 


ZCT  V^1  CE  =  0;  Z  =  cov  (eKp) 


C  =  pilot's  observation  matrix 


These  relations  are  all  obtained  from  Ref.  f5)  and  from  the  pilot  model 
equations  given  in  Ref.  (1).  Here  e^p  is  the  Kalman  filter  estimation  error 
for  the  delayed  state,  •  z  the  covariance  of  e„ct  and 

Kr 
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W  lO 


Also  W  is  the  covariance  of  the  plant  disturbance  w,  and  Vu  and  Vy  are 
motor  noise  and  measurement  noise  covariance,  respectively,  all  assumed 
known.  Now  the  P  equation  may  be  integrated  over  the  time  delay  t,  with 
the  initial  condition  on  P  from  P(0)  =  z,  the  Kalman  filter  error  covariance. 
Now,  since  the  predictor  has  the  property  that  E(x  eT)  =  0,  we  have 


E 


E(e  e^J  =  P 


So  then  the  terms  E(x  eT)  and  E(upeT)  are  available  from  P,  and  these  are 
required  to  form  M. 

Finally wit  can  be  shown  (Ref.  (5)),  pg.  331)  that  with  the  processes  w  and 
vy  uncorrelated  we  have  in  this  case 

E(xJ)  =  0 

E( vl> '  ?  Vu 


Returning  then  to  the  estimation  of  the  gains  (equation  I),  we  see  that 
all  the  terms  in  the  matrices  N^,  N^  and  M  may  be  calculated,  either  analytically 
or  from  the  measurements  of  x,  up  (and  up).  The  estimate  for  the  gain 
vector  is  then 


Note  finally  that  the  matrix  M  is  formed  from  two  matrices 

M  =  M  -  M 
x  cor 

where  the  MCQr  and  Ny  matrices  may  be  thought  of  as  corrections  added  to  a 

basic  least-squares  technique.  The  potential  importance  of  these  terms  (Mcor 

and  N  )  will  be  demonstrated  in  an  example  later. 
vu 
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The  algorithm  is  as  follows: 

1)  Select  noise  covariance  matrices,  W,  Vu»  and  Vy 

2)  Select  a  time  delay  x, neuromuscular  time  constant  x^  (or  matrix 

TN  =  9U1}* 

3)  Form  and  solve  for  Kalman  Filter  error  covariance  g. 


4) 

5) 

6) 

7) 

8) 

9) 


Solve  for  covariance  matrix  P(x),and  then  the  Efe  7T)  is  availabl 
(Note,  ail!  these  steps  may  be  accomplished  before  or  after  the 
experimental  data  is  obtained.) 

Perform  experiment  to  obtain  state  and  control  (and  control  rate) 
time  histories. 

T  T 

From  the  time  histories,  obtain  estimates  for  E(xx  ),  E(xUp), 


E(u  u  ),  E(xuJ)  and  E(u  uj) ,  or  the  matrices  Mv  and  N*  . 
PP  P  p  p  |  ,  x  u 

Identify  M  and  Nv  in  E(77  )  found  in  step  4. 


cor 

u 

Form  M  =  Mx  t  MCQr  and  determing 
■1 


9x' 

9u 

*  • 


est 


from  Equation  II. 


Check  gu  vs  Tfj  (selected  in  2  above)  and  iterate  (steps  2-8) 
again  as  necessary.  Note  now  that  selecting  affects  the 
solution  for  £  and  P(x),  along  with  the  effective  Vu  or 


e. 


=  t; 


-i 

N 


“eff 
while  selecting 


x  only  affects  P(x) 


in  the  procedure. 


Comparison  to  Classical  Results 

It  is  interesting  to  note  that  the  "corrections"  performed  by  Including 
Mcor  and  Nvy  are  qualitatively  related  to  an  identification  technique 
(discussed  in  Ref.  6,  and  elsewhere)  used  to  determine  the  human  describing 


590 


function  in  a  compensatory  task,  which  goes  back  to  the  development  of  the 

"crossover  model"  of  McGruer  et  al .  Shown  in  Figure  4  is  a  schematic  of  this 

situation,  showing  the  closdd-loop  tracking  of  some  commanded  Bc.  Measurements 

may  be  taken  of  6  (t),  e(t),  u  (t)  ,  and  ©{ t)  and  manipulated  in  the  frequency 
c  p 

domain  to  obtain  frequency  spectra 

=  up(ju,)/ec(jw) 

G2(jo))  =  £(ja>)/ec(ja)) 

Gg(ju)  s  Up(J»)/e(j«) 

Now,  in  this  model  the  pilot's  control  is  considered  to  consist  of 

two  parts,  one  correlated  with  . the  input  ec,  the  other  uncorrelated  with  the 

input.  The  latter  component  was  defined  to  be  "remnant."  Mathematically, 

u  (J«)  ■  Yp( jus)  e(ja))  +  r( jo>)  and  r(ju>)/ec( ju>)  -»•  0  in  effect. 

Block  diagram  manipulation  leads  then  to  the  desired  relation 

Yp(jo>)  =  Gx(  j(o)/G2(  ju>) 

rather  than  the  simpler,  and  incorrect,  expression  Yp{ joi)  =  Gg(ju>).  This 
was  due  to  the  presence  of  remnant  r(jo>)  in  the  measured  control  input,  and 
the  necessity  to  eliminate  it's  effect  by  defining  it  as  the  uncorrelated 
component  of  up,  and  using  this  property.  Comparing  to  our  control  law, 
transformed  just  for  discussion  purposes,  we  have 

u  (jw)  =  gj  x(ju>)  +  gj  Up(joj)  -  gj  TCjoj)  +  vu(jw) 

' - v - '  * - - - 

unmeasurable 

separately 

compared  to 

u_( jco)  =  Y  (joi)  e( jco)  +  rCjo>) 

P  P  ^ 
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unmeasurable 

separately 


CLASSICAL  RESULTS 


MEASURE  Up(s)  =  Yp  e(s)  +  r(s) 

Yp  =  G2(jw)/G2(jw)  /  G^(jw) 
WHERE  G^(s)  =  Up(s)/©c (s) 
G2(s)  =  e(s)/©c(s) 
G3(s)  =  uP(s)/e(s) 


Figure  4 


The  significant  difference  is  that  r(jw)  was,  in  effect,  discarded 
in  finding  Y  ,  but  g^e  is  not  uncorrelated  with  x  or  u  and  must  be  accounted 

p  X  P 

for  in  the  identification  problem. 

A  Numerical  Example 

To  evaluate  the  numerical  properties  and  the  sensitivity  to  the  a  priori 

selected  parameters  (Vy,  W,  Vu,  t)  a  fast  time  simulation  of  the  pilot  model 

equations  has  been  assembled,  and  the  simulated  control  task  is  shown  in1 

o 

Figure  5  .  As  shown,  the  task  is  that  of  pursuit  trackinq  with  11.7/s 
controlled  element  dynamics,  and  the  displayed  command  signal  is  filtered 
white  noise  with  the  filter  transfer  function  given  (ec(s)/w(s)) .  The 
state  vector  is  shown,  the  known  gain  vector  to  be  identified  is  listed,  and 
the  weights  in  the  objective  function  used  are  given.  A  sample  time  history 
of  the  state  and  simulated  pilot's  control  input  is  depicted  in  Figure  6. 

Such  time  histories  were  sampled  at  10  msec  intervals  and  the  gains  estimated 
from  time  windows  of  data  5,  10,  15,  20,  25,  30  and  35  seconds  wide.  The 
root-sum-squred  percent  error  of  the  five  estimated  gains  is  shown  in  Figure  7. 


Where 

ERSS  = 

[TpVH 

i=l 1  1  J 

"  M 

As  shown,  about  30  seconds  of  data  is  required  to  obtain  less  than  10%  rss 
error  in  this  example.  Other  dynamics  of  higher  order,  and  therefore  more 
gains,  will  be  evaluated  in  the  near  future  and  the  convergence  will  not 
be  as  rapid. 

The  importance  of  using  the  proper  corrections  (e.g.,  M  and  N  ) 

cor  u 

is  shown  in  Figure  8  ,  in  which  the  five  exact  gains.,  g^  g5  are  shown, 
along  with  two  sets  of  gain  estimates.  The  set  labeled  "uncorrected"  was 
obtained  via  straight-forward  least  squares  (i.e.,  MCQr  and  not  included). 
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EXAMPLE 


Plant: 


e(s) 

6lsT 


ULZ 

s 


0r(s) 

W- 


5,6.7 _ 

+  3s  +  2,25 


xT  -  [ec  9c  9  9] 

X  =  AX  +  BU  +  N 

Up  =  gJ  Xp  +  Gu  Up  +  Vy 

[GJ  Gu]-  [5.53,  1.86,  -6.76,  -3.69,  -9. 28^ 
qe  =  16/. 35,  q*  =  1/. 35,  Q.%  =  1 


Figure  5 


CONVERGENCE  RATE 
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Figure  8 


Conversely,  the  "corrected"  set  used  perfectly  corrected  data,  or  the  actual 

A 

X's  in  the  identification.  Both  sets  of  gain  estimates  are  based  of  50 
seconds  of  data.  Clearly,  in  this  case  again,  the  corrections  are 
important.  Further  verification  of  the  method  is  in  process. 


Inference  of  the  Objective  Function 

Attention  is  now  turned  to  estimation  of  the  objective  function  weightings 
from  the  gain  estimates  just  discussed.  (Note,  this  is  referred  to  in  the 
control  literature  as  the  "inverse  problem".)  These  weights  are 
related  to  the  gains  via  the  Riccati  matrix  K,  the  solution  of 


and 


where 


ATK  +  KA  +  Q  -  KBG-1BTK  =  0 


BT  =  fo;  I  1  I  =  identity  of  dimension  equal  .to  u  control 
L  J  u  vector  p 


And  recall  that  Q^,  R,  and  G  are  the  weightings  defined  in  Eqn  (0).  Now 
due  to  the  structure  of  the  OCM,  we  are  able  to  reduce  the  above  into  some 
simpler  relations.  First,  noting  that  letting  G  =  Iu,  without  loss  of  generality 
(at  least  in  the  case  of  scalar  control  input  Up),  we  obtain 


and 


R  = 


q  +  q^B 
au3u  sx 


+  b\ 


(III. a) 
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(Ill . b) 
(III .c) 


0  =  q  q  -  LB  +  A^q 
yxyu  yX 

cTQyc  =  gxgj  -  LA  -  atl 
Now  L  can  be  eliminated  in  the  last  two  relations  tf  desired,  and  have 

BTCTQ  CB  =  BTg Jb  -  H  -  HT  (IV) 

\y  ax3x 

where 

H  -  (g„gj  +  sJa)AB 

By  observing  Equation  III  (and  IV)  we  can  see  that  if  estimates  of  gx 
and  gu  are  available,  and  plant  and  observation  matrices  A,  B,  and  C  are 
known,  the  R  weighting  can  be  obtained  directly,  but  requires  special 
attention.  From  Egns.  Ill .b  and  .c  we  see  that  if  L  can  be  obtained  by 
solving  Ill.b,  an  n  x  n  matrix  equation  with  only  CTQC  unknown  results  from 
III  .c .  But  this  is  only  possible  if  B-'1  exists,  which  is  only  triue  if  the 
number  of  independent  control  inputs  (in  Up)  equals  the  number  of  states 
(in  x)-an  unlikely  situation. 

An  alternate  attack  using  Eqn.  IV  leads  to  similar  results.  One  could 
conceivably  solve  for  a  diagonal  via  a  numerical  method  like  Newton- 
Raphson,  but  that  requires  the  matrix  CBB^C^  to  be  inv.ertable.  This  is  possible 
if  the  number  of  control  inputs  (in  Up)  equals  the  number  of  outputs  (or  y), 

(or  the  system  transfer  function  matrix  is  square).  Although  this  is  less 
restrictive  than  the  previous  situation,  it  is  also  untrue  in  many  applications 
of  interest  to  us  here.  So  the  following  conclusions  may  be  stated,  that  in 
general  a  unique  set  of  objective  function  weights  may  not  be  obtainable  from 
gain  estimates  alone.  This  result  is  not  new,  we've  just  looked  at  it  in 
the  context  of  our  specific  problem. 
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Although  improved  methods  are  currently  under  investigation  in  this 
regard,  we  may  always  test  assumed  objective  function  weights  to  determine 
if  they're  feasible.  This  is  considered  a  reasonable  alternative  since  in 
an  actual  experiment,  the  analyst  knows  several  reasonable  statements  for 
the  objective  function,  and  he  may  at  least  test  them  to  see  which  one  is 
best  supported  by  the  data.  To  pursue  this  approach,  the  accuracy  of  the 
gain  estimate  will  also  be  developed  such  that  statistical  hypothesis  tests 
may  be  performed.  But  for  now,  this  is  an  important  consideration. 

In  the  case  of  our  numerical  example.  Equation  III. a  leads  to  R  =  0, 
and  Equation  IV  yields 


(ll.7)Z(q.  +q.)  =  -2(11.7)gugX3  +  (11.7  g  ) 


where 


0 

0 


0 

0 

% 

0 


0 

0 

0 


qe 


Using  the  estimated  gains  we  obtain 

(q*  +  q*)  =  2.89  (actually  q.  was  1/.35  and  q>  was  0) 


Now  "guessing"  that  qQ  and  q*  were  zero,  at  first,  we  may  iterate  on  q^ 
and  solve  III.c,  then  check  with  Ill.b.  If  no  q  >0  led  to  a  solution, 
then  the  assumption  of  qQ  and  q*  equal  to  zero  would  need  revision.  Finally, 
note  that  from  Equation  Ill.b,  we  can  actually  solve  for  as  many  columns  (and 
rows)  of  L  as  the  number  of  control  inputs  (or  rank  B),  and  this  part  of 
the  L  matrix  may  be  used  to  check  results  from  III.c. 
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ABSTRACT 


Validation  of  a  model-based  methodology  for  the  design  and  evaluation  of 
advanced  cockpit  information-display  systems  is  undertaken.  The  technique  is 
applied  to  determine  information  and  display  requirements  in  an  A10  terrain¬ 
following  flying  task.  Four  candidate  display  systems,  including  a  flight 
director  system,  are  proposed  and  rank-ordered  across  dimensions  of  workload 
and  performance.  Validation  of  the  analytical  predictions  is  accomplished 
through  man- in- the -loop  simulation  experiments.  It  is  concluded  that  the 
methodology,  which  can  determine  the  limit  information  to  the  best  quantities 
needed  by  the  pilot  to  perform  effectively,  can  be  a  valuable  tool  in  the 
development  of  advanced  cockpit  information  systems. 


INTRODUCTION 

The  increased  complexity  of  advanced  tactical  air-to-air/close  air  support  air¬ 
craft  has  elevated  the  pilot  to  a  role  of  systems  manager.  The  pilot  must 
allocate  sensors,  evaluate  threats,  select  weapons,  employ  ECM,  etc.,  in  addi¬ 
tion  to  the  omnipresent  task  of  flying  the  aircraft.  The  increase  in  the 
number  of  systems  and  subsystems  has  required  the  display  of  increasingly  more 
information  to  the  pilot  about  his  aircraft  and  the  environment.  Unfortunately, 
this  has  increased  the  pilot's  workload.  A  methodology,  therefore,  is  required 
to  determine  and  limit  information  to  the  best  informational  set  needed  by  the 
pilot  to  perform  his-  various  tasks  effectively. 

If  pilot  and  aircraft  are  to  function  as  an  effective  combat  system,  it  is 
essential  that  the  workload  associated  with  the  basic  flying  task  be  reduced 
through  either  display  or  control  automation.  Thus,  the  pilot  would  be  in  a 


*  This  work  performed  under  the  following  contract  to  AMRL:  F33615-80-C-0528 
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position  where  he  can  allocate  a  greater  fraction  of  his  capacity  for  decision 
making  relative  to  weapon  assignment,  etc.  Nowhere  is  this  problem  more 
critical  than  in  high-speed,  low-level,  adverse-weather  interdiction  where  the 
workload  demands  of  the  flying  tasks  are  severe. 

The  design  of  automated  control/display  systems  for  reducing  workload  in 
piloting  tasks  is  generally  accomplished  through  extensive  recourse  to  man-in- 
the  loop  simulations.  This  can  be  an  expensive  and  time-consuming  approach, 
especially  when  large  numbers  of  competing  designs  and/or  parameter  sets  must 
be  evaluated  and  iterated.  While  simulation  is  ultimately  necessary  in  the 
development  of  control/display  systems ,  it  would  be  desirable  to  have  a  computer- 
based  tool  that  could  perform  a  preliminary  evaluation  on  a  wide  variety  of 
systems  on  a  relative  performance  basis.  In  this  manner,  one  would  only  need 
to  retain  those  systems  exhibiting  promise  for  further  evaluation  by  manned 
simulation. 

A  model/computer-based  methodology  has  recently  been  proposed  [1-2]  that  offers 
considerable  potential  for  application  to  tactical  aircraft  display  systems 
design  and  evaluation.  Although  this  methodology  has  been  applied  to  rank- 
order  control/display  systems  for  a  CH-47[1],  the  design  technique  has  remained 
largely  unvalidated  due  to  the  lack  of  subsequent  manned  simulation  to  assess 
the  preliminary  analytic  results.  In  this  study  the  design  procedure,  as  pro¬ 
posed  by  Kleinman  and  Curry,  is  extended  to  a  practical  cockpit  scenario.  Sub¬ 
sequently,  we  validate  the  model  predictions  via  man-in-the-loop  simulation 
experiments.  The  piloting  control  task  considered  is  a  representation  of  a 
zero-visibility,  high-speed,  low-level  terrain  following  scenario  with  an  A10 
aircraft.  The  aircraft  dynamics  are  approximated  by  their  short-period  longi¬ 
tudinal  equations  in  the  analysis  and  simulation. 

The  focus  of  this  study  is  on  display,  design,  and  evaluation,  including  optimal 
synthesis  or  aggregation  of  information  states,  for  the  pilot  control  task.  The 
design  methodology,  when  applied  to  display  systems  evaluation,  follows  a  three- 
level  procedure.  At  the  first,  or  information  level,  the  methodology  determines 
the  relative  importance  of  each  system  variable  to  closed-loop  task  performance 
and  determines  the  optimal  synthesis  of  information  in  the  contex  of  a  flight 
director  signal.  At  the  second,  or  display  element  level,  the  information 
requirements  are  integrated  to  propose  several  different  realistic  display 
systems.  Human  generated  information  processing  limitations  are  included  at 
this  level,  and  for  each  candidate  design,  performance  versus  workload  curves 
are  generated.  The  different  display  systems  are  then  rank-ordered  across 
dimensions  of  performance  and  workload.  The  proposed  displays  that  resulted 
via  the  application  of  the  methodology  to  the  terrain  following  scenario 
included:  (1)  a  terrain  predictor/flight  path  vector  display,  (2)  a  tunnel 

display,  (3)  an  integrated  tunnel-predictor  display,  and  (4)  an  optimally 
designed  flight  director  display. 

The  display  format  level  is  the  third  and  final  phase  of  the  design  process. 

Here,  specific  display  formats  are  suggested  for  the  presentation  of  the  candi¬ 
date  display  systems,  guided  by  the  sensitivities,  attentional  demands,  etc., 
that  are  predicted  at  the  display  element  level.  The  format  level  is  the 
precursor  to  manned  simulation  experiments.  In  this  effort,  a  man-in-the-loop 
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experimental  program  was  conducted  at  the  University  of  Connecticut  to  evaluate 
the  four  candidate  display  systems.  The  objective  of  these  experiments  was  to 
validate  the  overall  display  design  procedure,  including  the  analytic  rank¬ 
ordering,  the  workload  assessment,  and  the  flight  director  design  synthesis 
processes. 

The  analytical  nucleus  to  the  display  design  methodology  is  the  Optimal  Control 
pilot  Model  (OCM) [3]-[5],  A  detailed  description  of  the  OCM,  in  conjunction 
with  the  three-level  display  design  process,  is  given  in  [6],  In  this  paper 
we  first  introduce  the  control  task  of  interest,  including  the  A10  aircraft 
dynamics  and  the  terrian  profile  characteristics.  We  apply  then  the  display 
design  procedure,  which  yields  the  major  analytical  results  of  this  study, 
including  the  predictions  of  attentional  allocation,  workload  demands,  and 
performance  rank-ordering.  Finally,  the  experimental  program  is  described, 
and  the  experimental  results  discussed. 


APPLICATION  OF  THE  CONTROL  TASK  TO  THE  OCM 
A-10  LONGITUDINAL  AIRCRAFT  DYNAMICS 


As  indicated,  the  control  task  of  interest  is  high-speed  terrain  following  for 
a  representative  low-level  attack  flight  condition.  The  basic  set  of  longi¬ 
tudinal  equations  being  used  is  the  short-period  dynamics.  These  are  the 
perturbation  equations  written  about  straight  and  level  flight,  and  they 
describe  the  aircraft  longitudinal  rotations  about  its  center  of  mass. 

The  (two-degree  of  freedom)  transfer  functions  of  interest  are  [7] 


a 

6 


(s) 


.  Z,s  +  (U  M--Z,M  ) 

_ 0 _ o  o  o  q 

U  A  (s) 

o 

(M,+Z ,M* ) S  +  (Z.M  tMxZ  ) 
6  6  w _ 6  w  6  w 

A  (s) 


(1) 

(2) 


A (s)  =  s2  -  (U  M*+Z  +M  )s  +  (M  Z  -U  M  )  (3) 

owwq  qwow 

where  q  =  aircraft  pitch  rate,  a  =  angle  of  attack,  U  =  nominal  air  speed, 

6  =  elevator  angle,  and  Z~,  M . ,  M  ,  M* ,  Z  are  the  pertinent  stability 
.  .  .  .  o  o  q  w  w 

derivatives . 

In  the  present  work  the  stability  derivatives  were  derived  from  data  currently 
used  on  the  Aerospace  Systems  Division  A-10  nonlinear  hybrid  simulation,  and, 
the  numerical  values  are  given  in  [6] .  The  nominal  air  speed  Uq  =  468  ft/sec. 

In  addition,  the  aircraft  pitch  angle,  0,  and  its  altitude,  h,  are  necessary  in 
the  modeling  process.  Therefore 

0  =  q  ;  h  =  U^ (0-q)  (4) 
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TERRAIN  PATH  MODEL 


The  next  step  is  to  select  an  appropriate  terrain  path  model  that  serves  as  the 
excitation  "signal"  to  the  system.  An  easily  implemented  one  dimensional  model 
for  the  terrain  height  II  as  a  function  of  time  consists  of  a  Markov  process 
passed  through  a  2nd  order  filter*  [6].  The  Markov  process  is  generated  by 

2U  2U 

2(t)  +  z(t)  =  S(t)  (5) 


where  £(t)  is  a  white  Gaussian  noise,  and  X  represents  the  spatial  terrain 
variations.  Next,  z(t)  is  filtered  through 


H  (s ) 


0)2U2 


s2+2£o)U  s+to2u2 
o  o 


(6) 


where  to  =2tt/D  is  the  natural  frequency  of  the  terrain  (D  =  "terrain  period") 
and  £  is  the  damping  ratio.  In  order  to  represent  a  realistic  terrain  profile, 
the  terrain  parameters  are  selected  appropriately  [6] .  The  parameter  values 
chosen  to  give  the  most  appealing  terrain  characteristics  where 

D  =  10000'  ;  X  =  D/it  ;  ?  =  1/^J~2 

The  white  noise  (£(t))  intensity  was  chosen  to  yield  a  reasonable  terrain  RMS 
value  of  ~136  ft. 

OCM  APPLICATION 

The  aircraft  dynamics  and  the  terrain  states  are  combined  into  a  single  state 
equation  as  required  by  the  OCM.  The  augmented  state  is  defined  as 

x'  =  [n  n  H  a  q  0  e]  (7) 

where  e(t)=  flight  path  error  =  H(t)  -  h(t).  The  total  state  space  equation 
is  now 

x (t)  =  Ax (t)  +  B6  (t)  +  E£(t)  (8) 


The  term  II  should  represent  a  spatial  rather  than  temporal  terrain  model.  How¬ 
ever,  assuming  a  constant  velocity,  U  ,  and  neglecting  the  flight  path  flactua- 

tions,  we  can  write  n(r)~II(U  t)=U  II (t?,  where  rc:U  t  represents  the  traveled 

, .  .  o  o  o 

distance. 
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The  numerical  values  of  A,  B,  E  are: 


A  = 


0 

0 

-.025 

0 

0 

0 

0 


1 

0 

-.21 

0 

0 

0 

1 


0 

1 

-.71 

0 

0 

0 

0 


0 
0 
0 

-1.37 
-3.08  -1.68 
0  1 
8.17  0 


0 

0 

0 

1 


0 

0 

0 

0 

0 

0 

-8.17 


0 

0 

0 

0 

0 

0 

0 


B’  =  [0  0  0 
E'  =  [0  0  .025 


-0.06  -6.84  0 

0  0  0 


o] 

0] 


The  terrain  following  task  requires  the  pilot  to  follow  the  path-over-terrain 

II(t),  as  closely  as  he  can,  subject  to  his  inherent  limitations.  Thus,  the 

quantity  he  should  minimize  is  the  flight  path  error  (FPE)  e(t).  In  addition, 

the  pilot  should  avoid  large  vertical  acceleration  values,  or  g's.  These 

requirements  on  e (t)  and  g(t)  (or,  equivalently  on  q(t))  are  expressed  in  the 

cost  functional  associated  with  the  OCM,  with  the  pilot's  subjective  weightings 

q  and  q  ,  viz. , 

^e  q 

J(6,f)  =  E{qge2(t)  +  qgq2(t)+  q^62(t)}  ,  (9) 

where  the  control  rate  weighting,  qj,  is  determined  by  the  pilot’s  neuromuscular 
time  constant  x^  [3] -[6].  The  cost  functional  J(6,f)  is  minimized  with  respect 
to  the  control  6  and  the  attentional  allocation  vector  f . 
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The  nominal  altitude  of  the  trajectory  above  the  terrain  (II  ( t ) )  the  aircraft 
must  follow  is  200  ft. ,  and  we  assume  that  the  maximal  excursion,  or  FPE,  should 
be  no  more  than  20%,  or  emax  =  40  ft.  Also,  we  assume  that  the  maximal  vertical 
acceleration  to  be  tolerated  by  the  pilot  is  az  max  =  Ig  =  32.2  ft/sec2.  Thus, 
we  compute  the  FPE  and  pitch  rate  weighting  according  to  [1] ,  [2]— [6] ,  viz., 


^  180°  '  z ,max  , 

q  S -  •  — n -  =  4°/sec 

Tnax  7r  U 

o 

q  =  — =  6.25  •  10~4ft  "2 
e  e  2  4q2 

max 


<I„  = 


Sn. 


— —  =  =  6.25  •  10~2  deg-2  sec2 


The  assignment  of  the  neuromuscular  time  constant  tn  =  .15  sec  completes  the 
specification  of  the  cost  functional  Eq.  (9) .  It  must  be  indicated  that  J 
is  the  "nominal"  cost  functional.  With  new  display  systems  incorporated  in  the 
information  set,  it  is  possible  that  the  pilot  will  attempt  to  minimize  addi¬ 
tional  indicators.  This  will  be  explained  in  detail  in  the  sequel. 


DISPLAY  SYSTEMS  DESIGN 

We  present  now  the  major  analytical  results  of  the  display  systems  design  for 
the  candidate  terrain-following  task. 

INFORMATION  LEVEL  ANALYSIS 

The  information  level  analysis  is  used  to  determine  information  requirements  in 
the  preliminary  stages  of  display  design  and  is  the  cornerstone  to  the  sub¬ 
sequent  display  level  analysis,  There  are  two  basic  tasks  at  the  information 
level:  the  information  requirements  assessment  and  the  flight-director  design. 

Information  Requirements  • 


At  this  step  we  assume  that  all  state  variables,  and  possibly  their  rates  are 
accessible  to  the  pilot.  Only  the  plausible  rate  variables  should  be  considered. 
After  some  preliminary  analysis,  the  following  set  of  variables  was  explicity 
considered : 


y'  =  [n  n  n  e  e  h  h  9  q]  (10) 

In  the  OCM  application  we  use  the  cost  functional  defined  by  Eq.  (9) .  The 

resulting  terrain  following  performance,  in  terms  of  absolute  values  of  the 

flight  path  error,  is  not  of  immediate  interest  at  this  level.  We  are 

interested  in  the  relative  importance  of  each  state  variable  to  the  control 

task.  Thus,  the  performance  index  (9)  is  minimized  subject  to  Ef^  =  fc»  f c^0 , 

where  f  .  is  the  attentional  allocation  to  the  ith  indicator.  This  procedure 
ci 


607 


is  repeated  for  several  values  of  control  attention,  fc,  and  the  summation  is 
taken  over  the  nine  elements  of  Eq.  (10) .  The  optimal  fci  that  ensue  provide 
the  relative  importance  of  each  of  the  states,  i.e.,  the  information  require¬ 
ments  for  the  control  task. 

The  OCM  analysis  yields  the  relative  attentional  allocation  (fci/fc) ,  summari¬ 
zed  in  Table  1  and  Figure  1.  The  attentional  allocations  are  examined  at  four 
levels  of  total  control  attention,  fc  =  0.9,  0.7,  0.5,  0.3.  The  only  variables 
that  command  significant  pilot  attention  are  the  'terrain  "vertical  acceleration" 
K (t) ,  the  FPE  rate,  e(t),  and  the  FPE  e (t) .  All  other  system  states  demand 
negligible  attention  in  the  pilot  model. 


TABLE  1.  RELATIVE  ATTENTIONAL  ALLOCATION-  f 


INFORMATION  LEVEL 

cl 

—  f 

c  0.6. 

f 

c 

f  n/fc 

f-/f 

e  c 

f  /f 
e  c 

O.ii . 

- * - 

•  iT 

0.9 

0.46 

0.34 

0.1 

0.2. 

r— ' 

0.7 

0.46 

0.  36 

0.1 

• . -.".-4  #  . 

*  e 

0.5 

0.42 

0.  36 

0. 1 

o'.  2  o'.  1*  0.6  0.8 

l.'o 

0.3 

0.37 

0.33 

0.1 

i  .i.m---.  ■.  ■  -■■■  -  *■■■■  ■  ■■■■■-  . —  —  Figure  1.  Relative  Attentional 

Allocation-Information  Level. 

Notice  that  there  is  a  consistency  in  these  results,  i.e., 

f  f"  £' 

e  II  e 

—  =  const.  ,  —  £?  const.  ,  =  const. 

c  rc  c 

Thus,  the  relative  importance  of  the  key  state  variables  does  not  change  over 
different  levels  of  control  attention  fc. 

The  results,  which  indicate  that  n(t),  e (t) ,  and  e(t)  is  the  critical  informa¬ 
tion  base,  must  now  be  interpreted.  A  major  design  issue,  at  this  point,  is 
in  determining  whether  or  not  a  separate  display  of  each  critical  variable  is 
required  for  use  by  the  pilot.  It  is  clear  that  a  FPE  indicator  should  be 
included  since  it  is  essential  that  the  pilot  knows  his  instantaneous  position 
with  respect  to  the  desired  path  JI(t).  Such  a  display  is  easily  constructed 
using,  e.g. ,  a  radar  altimeter.  Also,  the  pilot  can  easily  extract  the  FPE 
rate  from  a  well  designed  display  of  e  (t) ,  and  therefore,  a  separate  e (t) 
indicator  is  not  needed. 

The  major  portion  of  the  pilot's  attention  is  allocated  to  II (t).  In  practice, 
this  variable  is  difficult  to  derive  and  display  explicity.  Also,  it  is  not 
clear  how  such  a  display  might. .be  used  by  the  pilot.  One  must,  therefore, 
synthesize  a  display  in  which  II (t)  will  be  incorporated  in  a  practical  manner. 


608 


One  interpretation  of  the  requirement  for  rate  information  IHand  e)  is  the 
necessity  to  display  the  path-over-terrain  at  a  future  time,  i.e.,  at  some 
distance  ahead.  Such  displays  will  be  synthesized,  modeled,  and  compared  at 
the  display  element  level. 

Flight  Director  Design 

The  basic  concept  of  a  flight  director  is  to  provide  the  pilot  with  (synthesized) 
information  that  is  useful  for  control,  thus  rendering  the  piloting  task  easier 
in  some  sense.  This  section  describes  the  flight  director  design  procedure 
that  uses  the  quadratic  synthesis  technique  of  the  OCM  to  determine  the  optimal 
(linear)  aggregation  of  system  states  to  be  used  as  a  display. 

The  flight  director  signal  is  a  linear  combination  of  the  system  states, 


yFD(t)=h'x(t)  (11) 

The  gains  h  are  chosen  so  that  if  y  (t)  is  kept  "small"  by  the  pilot,  the 
resulting  aircraft  motion  will  be  desirable.  Since  the  pilot  is  in  control  of 
the  vehicle,  there  are  two  issues  that  relate  to  the  harmony  between  yF£,(t)  and 
pilot  response.  The  first  concerns  the  nature  of  the  control  task  as  viewed 
by  the  pilot.  Thus,  the  task  of  keeping  yFD(t)  small  should  not  conflict  with 
the  overall  pilot-control  task  requirements.  The  second  issue  relates  to  the 
required  form  of  the  pilot  compensation,  as  yFD  and  6  are  in  one-to-one  corre¬ 
spondence.  From  a  reduced  workload  point  of  view,  one  should  design  a  flight 
director  signal  yFD(t)  such  that  the  transfer  function  from  input  6(t)  to  yFD(t) 
is  approximately  K/s.  The  required  pilot  compensation  would  then  be  simple  pro¬ 
portional  feedback 

fi(t)«K  •  yFD(t)  (12) 

From  the  OCM,  the  pilot's  control  strategy  is  given  by 

T  6  +  6  =  -Lx (t)  +  v6(t)  (13) 

where  x(t)  is  the  state  estimate,  v^ft)  is  a  white  motor  noise,  and  the  gains 
L  (and  Tn)  are  obtained  by  minimizing  the  cost  function  J(6)  that  is  associated 
with  the  terrain  following  task,  Eq.  (9).  To  begin  with,  we  suggest  the  obvious 
design  choice 

yFD(t)  =  Lx(t)  ,  (14) 

i.e.,  h'  =  L.  Such  a  design,  however,  does  not  consider  the  possibility  that 
the  flight  director,  once  added  to  the  display  panel,  modifies  the  pilot's 
control  task  and  hence  changes  the  cost  functional  J(6J.  Excluding  ypp  from 
the  cost  functional  implies  that  the  pilot's  control  objectives  are  basically 
the  same  as  before  introducing  this  signal.  This  is  not  a  reasonable 
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assumption.  Indeed,  including  the  yFD  with  J(6),  in  addition  to  the  other 
terms,  implies  that  one  of  the  pilot's  direct  control  objectives  is  to  keep 
the  yFD  small.  Thus,  we  assume  that  the  director  signal  yFD  is  explicitly 
controlled. 


The  control  cost  functional,  modified  to  weight  deviations  of  yF£,(t)  is  now 


J’ (6)  =  J(«)  +  E  ^qFDyFD2(t) 


(15) 


The  weighing  term  qFD  is  selected 


FD  y. 


1 

T 


(16) 


FD,max 


to  be  consistent  with  the  choice  of  the  qyi's.  The  maximal  flight  director 
excursion  is  computed  according  to  the  rule 


y  =  ^  \ .  1 1. |x. 

*FD  ,max  i1  i1  i. 


max 


(17) 


where  the  1^  are  the  entries  of  the  gain  vector  L,  and  the  y±  is  0  or  1  to 
indicate  which  variables  are  of  concern  in  forming  yFD,max*  We  select 

11  if  X-:  is  a  positional  variable 

.  ,  (18) 

0  if  Xj^  is  a  rate  variable 

Thus,  the  flight  director  signal  is  at  its  maximum  value  when  all  error  dis¬ 
placements  are  at  their  design  limits. 

With  the  pilot  cost  functional  modified  as  in  Eq.  (15) ,  the  pilot  model  control 
is  now  obtained  by  minimizing  a  new  cost  functional,  the  result  being 

6  +  6  =  -Lx(t)  +  v.(t)  (19) 

N  o 

But  since  the  cost  functionals  of  Eq.  (9)  and  (15)  are  not  the  same,  the  opti¬ 
mal  gains  L  in  Eq.  (19)  differ  from  those  in  Eq.  (13).  Hence,  the  flight 
director  signlas  of  Eq.  (14)  and  the  required  pilot  control  gains  in  Eq. . (19) 
are  no  longer  in  harmony.  This  mismatch  can  be  corrected  via  the  iterative 
process  of  computing  feedback  gains  and  flight  director  signals  as  shown  in  (1] . 

This  algorithm  has  given  rapid  convergence  in  the  terrain  following  problem. 

Only  four  iterations  have  been  needed,  and  the  resulting  converged  gains  were 
within  10  percent  of  the  initial  gain  values  obtained  from  Eq.  (9) . 
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The  numerical  results  of  the  flight  director  design  process  are: 


1. 

design  parameter,  yFD,max  — 3 

2. 

flight  director  weighting,  q  =  -  2  - -  = 

32  “-1 

3. 

flight  director  signal  FD,max 

yFD  =  Lx  =  H  3-10~3  .13  .11  .76  -.48 

-1.2  .06 J  x 

We  do  not  yet  examine  terrain  following  performance  with  the  flight  director 
display,  as  this  is  the  subject  of  the  display  element  level  in  wich  all  pro¬ 
posed  displays  are  compared, 

DISPLAY  ELEMENT  LEVEL  ANALYSIS 


The  next  step  in  the  display  design  methodology  is  to  evaluate  control  per¬ 
formance  for  several  candidate  display  systems.  The  evaluation  process  begins 
with  the  definition  of  a  performance  metric  for  the  terrain  following  task. 


Often,  the  control  performance  requirements  of  the  pilot-vehicle  combination  are 
specified  in  terms  of  allowable  excursions  or  desired  RMS  deviations  in  system 
states.  The  design  specifications  are  generally  a  function  of  mission  require¬ 
ments  or  flight  conditions.  Recall  that  the  nominal  altitude  of  the  aircraft- 
over  the  terrain  is  200',  and  that  the  maximal  FPE  deviation  "allowed"  is  40'. 
Thus,  emax  =40'  is  chosen  as  the  design  tolerance  in  the  control  performance 
metric,  which  is  defined  as 


exp 


e2  (t) 

2<j2 

e 


d  [e  (t)  ] 


(20) 


This  performance  metric  measures  the  probability  that  the  aircraft  does  not 
deviate  more  than  emax  =  +40'  from  the  desired  path  II  (t) .  The  maximum  level 
of  control  performance  is  selected  as  Pc>max  =  .99. 


The  OCM  parameters  to  be  used  in  all  candidate  display  systems  are: 


observation  noise  (for  all  eventual  indicators),  Pyi  =  .20  dB; 
motor  noise,  =  -25  dB;  time-delay,  Tp  =  .15  sec. 

The  neuromuscular  time-constant  has  already  been  specified  as  tn  =  -15  sec. 


Using  these  parameter  values  and  the  performance  metric  Pc,  Eq.  (20) ,  we  now 
evaluate  and  compare  the  candidate  display  systems. 
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Display  System  Synthesis 


STATUS  DISPLAY 

The  status  display  system  consists  of  a  FPE  e(t)/e(t),  and  pitch  and  pitch  rate, 

0  (t) ,  q(t),  indicators.  This  rudimentary  display  is  not  a  truely  synthesized 
system  at  which  this  study  is  aimed;  nevertheless,  we  use  the  status  display 
as  a^benchmark  in  the  display  system  comparison  process.  The  selection  of 
[e,  e,  0,  q]  for  the  status  display  set  is  natural  [6],  and  all  subsequent 
display  systems  include  the  status  display  set.  Naturally,  the  status  display 
system  alone  should  yield  the  worst  terrain  following  performance.  Also,  we 
assign  indifference  threshold  values,  a^  to  e  (t) ,  e(t),  0 (t) ,  and  q(t) , 
according  to  the  rule  [6]  a^  =  [y^maxl/8-  Following  the  usual  assumption  that 
rate  variable  thresholds  are  one  half  of  the  thresholds  on  the  corresponding 
position  variables ,  we  obtain 


a 

e 


1 

■S'  e 
8  max 


1 

a 

2  e 


2. 5' /sec  ; 


a 

q 


1 

8  %ax 


1 

8 


»4°/sec 


0.5°/sec;  a  =  2*a  =1° 
0  q 


PREDICTOR  DISPLAY 

Grunwald  and  Merhav  have  shown  [8}-[9]  that  acceleration  cues  are  vital  in 
visual  field  control  and  that  they  are  obtained  by  estimating  the  future  vehicle 
path.  These  findings  are  in  close  agreement  with  our  information  level  analysis, 
in  which  it  was  shown  that  the  terrain  "acceleration",  H(t),  demands  the  largest 
amount  of  attention  among  all  system  states.  Such  information  can  best  be 
derived  (or  estimated)  by  the  pilot  when  the  vehicle’s  future  altitude  is  dis¬ 
played  relative  to  the  future  terrain-path.  We  define  T  as  the  prediction  time 
(i.e.  T~D0/U0,  where  Dp  is  the  distance  ahead  at  which  the  prediction  is  dis¬ 
played)  ,  and  the  predictor  signal  is  then  given  by 


e  (t)  =  II(t+T)  -  h  (t+T) 
P 


(21) 


The  underlying  assumption  in  the  predictor  display  is  that  the  predictor  vehicle 
path  is  along  the  velocity  vector  Up.  We  rewrite  Eq.  (21)  to  reflect  this 
assumption,  viz., 

7TTU 

e  (t)  =  n(t+T)  -  [h(t)  +  Th(t)]  =  II(t+T)  -  h(t)  -  [0  (t)  -a  (t)]  (22) 


Also,  the  OCM  assumes  that  the  pilot  derives  rate  information  from  the  predictor 
indicator,  e^(t).  The  appropriate  equation  for  this  signal  is 


e  (t)  =  H(t+T)  -  h(t)  -  Th(t) 
P 


ttTZ. 


.  7TU_  TTU 

n(t+T)  "lio?  6(t)  +w  l>TZJ  »<«  +  I5o=-  5(t> 


(23) 
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Such  a  display  is  easy  to  simulate  and  implement  using  e.g.,  a  forwardlooking 
radar.  An  actual  display  that  was  used  in  the  experimental  validation  is  shown 
in  Figure  6.  It  is  important  to  indicate  that  the  projection  of  U0  on  a  normal 
surface  located  T  seconds  flight  time  ahead  is  required  (represented  by  the 
cross  in  Figure  6. 

Equations  (22- (23)  in  their  present  form  cannot  be  modeled  directly  in  the  OCM 
steady-state  analysis,  where  only  events  at  time.t  are  treated.  This  problem  can 
be  "solved"  by  replacing  the  signals  II(t+T)  and  n(t+T)  with  their  (optimally) 
predicted  future  values,  viz., 


mt+T)  =  Ejn(t+T)|  n (t) ,  n(t),  n(t)j  ;  n (t+T)  =|e  n(t+T)|  n(t)  ,ii(t)  ,ii(t)j 

(24) 


Such  an  approximation  can  be  easily  obtained  from  the  3x3  upper-left  block 
of  A  in  Eq.  (8) .  This  results  in  estimates  which  are  a  linear  combination  of 
the  terrain  states  II  (t)  ,  ft(t)  ,  II  (t)  [6],  Specifically  we  may  write 


n(t+T)  = 

A 

ii(t+T)  = 


Pn(T)H(t)  +  p»(T)n(t)  +  p”(T)II(t) 
rn(T)II(t)  +  r^(T)H(t)  +  r“(T)II(t) 


(25) 


where,  as  indicated,  the  p  and  r  coefficients  are  a  function  of  the  prediction 
time,  T.  Thus,  we  rewrite  Eqs.  (22)-(23)  as  a  linear  combination  of  the  system 
state : 


ep(t) 

yt> 


=  C  (T)x(t)  +  D  (T)6(t) 
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180° 

_ 

(26) 


The  indifference  thresholds  on  the  predictor  are  assumed  identical  to  those  of 
the  FPE,  e (t)/e (t) . 


The  last  issue  which  must  be  addressed  is  the  selection  of  the  prediction  time 
T.  The  present  A10  HUD  uses  the  value  T^;4  sec.  This  value  has  been  selected 
for  use  in  the  present  display  analysis  and  in  the  subsequent  experiments.  It 
has  been  shown  [6],  however,  that  the  optimal  prediction  time  for  the  given 
control  task  and  assumed  terrain  characteristics  is  T*  =  1.5  sec.  The  numeri¬ 
cal  values  of  Cp(T)  and  D  (T)  of  Eq.  (26)  for  T  =  4  sec.  and  T=T*=1.5  sec.  are 
given  in  the  Appendix. 
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THREE  VJMENSJONAL  PERSPECTIVE  TUNNEL  VJSPLAV 


The  idea  of  a  displaying  3-D  perspecitve  tunnel  which  envelopes  the  trajectory 
over  the  terrain  is  not  entirely  new.  It  has  recently  been  studied  and  simula¬ 
ted  by  Grunwald  in  a  helicopter  approach  context  [10] .  Such  a  display  is  a 
computer-generated,  "through-the-windshield"  perspective  view  of  a  tunnel  that 
follows  the  contours  of  the  terrain.  In  practice,  it  seems  as  if  the  tunnel 
was  flying  towards  the  observer.  The  pilot,  on  his  part,  tries  to  maintain 
the  aircraft  as  close  to  the  tunnel's  center  as  possible.  Figure  7  shows  the 
tunnel  geometry  and  display  which  was  used  in  the  experimental  validation. 

The  present  FPE ,  e(t),  is  not  explicity  available  to  the  pilot  from  the  tunnel 
display.  He  can,  however,  derive  sufficient  rate  and  acceleration  information, 
as  dictated  by  the  information  level  results,  from  a  continuance  of  future 
flight  path  errors  displayed  by  the  perspective  tunnel.  It  is  necessary  now, 
to  translate  the  information  provided  by  the  tunnel  into  an  analytical  model 
for  application  of  the  OCM. 


A  plausible  approach  to  the  modeling  problem  has  been  suggested  by  Tomizuka 
[11]  in  the  so-called  "finite  preview"  problem.  If  the  tunnel  is  sufficiently 
long,  we  may  assume  an  infinite  preview  time.  The  finite  preview  problem  is 
then  reduced  to  a  common  optimal  tracking  problem,  and  can  be  treated  as  such. 

Both  approaches,  albeit  plausible,  require  major  modifications  in  the  OCM 
methodology  and,  therefore,  are  not  considered  here.  The  modeling  approach 
taken  in  the  present  study  treats  the  tunnel  display  in  the  OCM  framework.  As 
indicated,  the  tunnel  provides  a  continuance  of  future  flight  path  errors, 
assuming  a  straight  flight  path.  Formally,  this  information  base  may  be  re¬ 
presented  as  e(t+T),  xe[o,tp]  where,  in  general,  tp<  .  However,  such  a 
representation  is  impractical  from  a  modeling  point  of  view.  One  approxima¬ 
tion  is  to  replace  e(t+x)  with  Np  "indicators”  which  represent  the  FPE  values 
at  distinct  points  in  the  future,  viz.,  e(t+ai),  =  tpi/Np  where  i  varies 
from  1  to  Np.  The  indicators  e(t+a^)  would  then  be  treated  as  independent  Np 
predictor  display  systems.  Such  a  model,  although  requiring  an  extensive  set 
of  equations,  if  Np  is  chosen  to  be  large,  can  be  implemented  in  the  OCM  frame¬ 
work.  This  modeling  approach  is  carried  further  by  assuming  that  the  pilot 
concentrates  on  a  single  distance  ahead,  i.e.,  Np=l.  Since  the  underlaying 
assumption  of  the  OCM  is  that  the  pilot  adopts  an  optimal  control  policy,  it 
is  equally  valid  to  assume  that,  in  the  tunnel  display,  he  choses  the  optimal 
prediction  distance/time,  T*,  when  looking  down  the  tunnel  path.  We,  therefore, 
replace  the  Np  e(t+o^)  observations  with  a  single  indicator  e(t+T*).  The 
tunnel  display  model,  K(t),  is  then  simply 


[tc(t)l  re(t+T*)“| 

K(t)J  ■  [e(t+T.)J-  =„<**>««=>  +  yT*>{<«  <27 

where  the  numerical  values  of  Cp(T*)  and  Dp(T*)  T*=1.5  sec.  are  given  in  the 
Appendix. 
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Despite  initial  similarities,  there  is  a  fundamental  difference  between  a  simple 
optimal  predictor  display  where  T=T*=1.5  sec.,  and  a  tunnel  display.  Although 
both  displays  are  represented  by  the  same  equation  (26) ,  the  tunnel  display  does 
not  include  the  velocity  vector’s  tip,  U0  (Fig.  6)  projected  on  the  normal  sur¬ 
face  located  T*  seconds  flight  time  ahead.  The  end  point  of  UQ  is  estimated 
rather  than  displayed  in  the  perspective  tunnel.  .This  fact  is  reflected  in 
the  OCM  by  large  observation  thresholds  on  K(t),  K(t).  Given  the  +40'  tunnel 
dimensions,  as  implemented  in  the  subsequent  experiments,  we  assume 

aK  =  20’'  aK  =  1  aK  =  10’/sec 

These  thresholds  are  significantly  larger  than  the  indifference  thresholds  used 
for  the  simple  predictor  display  (5  ft  and  2.5  ft/sec  respectively).  Naturally, 
such  large  thresholds  tend  to  degrade  terrain  following  performance.  To  over¬ 
come  this  problem  the  following  display  system,  is  suggested. 


INTEGRATED  TUNNEL/ VELOCITY  VECTOR  VJSPLAY 

In  this  system  we  simply  superimpose  the  velocity  vector's  (UQ)  trace  on  the 
existing  tunnel  display  as  shown  in  Fig.  8.  Again,  the  value  used  is  T=4  sec 
and  not  T*,  in  accordance  with  the  current  A10  display  system.  It  is  obvious 
that  incorporating  this  new  information  will  reduce  the  thresholds  aK/a£,  since 
£he  pilot  now  has  a  reference  point  about  which  he  will  "center”  JI(t+T*).  Since 
UQ  is  projected  at  T=4  sec.,  and  the  pilot's  "focus"  in  the  tunnel  is  at  T=1.5 
sec.,  the  thresholds  aK/a £  are  not  reduced  to  the  ae/a^  values,  as  in  the  pre¬ 
dictor  display,  but  rather  to  an  intermediate  value.  We  select 

a  =  10' ,  a  =  5 '/sec 
k  K 

The  information  base  now  includes  both  tunnel  K(t)/ic(t),  and  a  4  second  pre¬ 
dictor,  ep(t)/ep(t),  (in  addition  to  the  status  display).  The  ep,ep  threshold 
values  remain  unchanged,  5',  2.5'/sec.  respectively. 


FLIGHT  VIRECTOR  SYSTEM 


The  status  information  base  is  now  augmetned  with  the  flight  director  position 
and  rate  observation  yFD(t) ,  yFp(t)  as  discussed  previsouly.  The  ypp/^FD 
observation  equations  are  given  by 


yFD(t)  =  Lx (t)  =  [3 . 10-3  .13  .11  .76  -.48  -1.2  .06]  x(t) 
yFD(t)  =  Lx(t)  =  LAQx(t)  +  LBQ6(t) 

=  [-3.10-3  .03  .05  .91  .31  -.48  0]x(t)  +  3.26(t) 


Also,  using  the  fact  that  y  ~3,  the  indifference  thresholds  are 

FD,max 

aFD  =  8  >yFD,max  =  8  *  3~*4'  aFp  =  2  aFD~  * 2 


(28) 

(29) 
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Implementation  of  the  flight  director  in  a. practical  manner,  using  future 
terrain  path  information  in  lieu  of  IT  and  H ,  is  described  in  the  sequel. 

Thus,  we  have  proposed  and  obtained  analytical  models  for  five  candidate  dis¬ 
play  systems.  The  next  task  in  the  analysis  procedure  is  to  evaluate  control 
performance  and  attention  allocation  for  these  systems. 


Control  Performance:  Modeling  Results 


The  performance  of  each  of  the  display  systems  is  evaluated  in  terms  of 

1.  control  performance,  and 

2.  acceleration  stress  levels 


Following  [1]  —  [6] ,  we  introduce  now  the  concept  of  control  and  monitoring  work¬ 
load.  The  control  workload  metric  is  based  on  the  fractional  attention  the 
pilot  allocates  among  the  various  display  indicators.  It  is  assumed  that  a 
pilot  distributes  a  total  amount  of  attention,  or  workload,  fT«0.8<1.0 
between  the  tasks  of  control  and  monitoring,  leaving  about  20  percent  of  his 
capacity  for  other  duties  (e.g.,  communications).  Let  fc  and  denote , 
respectively,  the  control  and  monitoring  attentions,  or  workloads.  Thus, 
fc  +  fm  =  fT-  The  attention  allocated  for  control,  fc,  is  distributed  among 
all  of  the  display  variables  y^,  y2,  ...»  yNy,  where  yi  and  yi+1  =  y^  (i  =  odd) 
are  obtained  from  the  same  display  indicator.  If  fci>  0  is  the  attention 
allocated  to  y^  for  control  purposes ,  then  the  constraints  on  fc£  are 


E 

i=odd 


f  . 
ci 


f 

c 


c,  i+1 


f  . 
ci 


i  =  1,  3,  5, 


(30) 


The  pilot  allocates  his  attention  among  displays ,  spending  the  larger  fc^  on 
displays  that  are  most  useful  for  control. 


With  fc;[  selected,  the  pilot-vehicle  model  yields  predictions  of  the  perfor¬ 
mance  metric,  Pc,  Eq. (20) .  Using  this  prediction,  we  can  study  the  tradeoffs 
between  fc  and  Pc  for  any  given  display  system.  Figure  2  is  a  typical  per¬ 
formance/workload  curve.  It  shows  the  performance  attained  for  a  given  work¬ 
load,  as  well  as  the  workload  required  to  obtain  a  given  performance  level. 

In  Figure  2,  the  intersection  of  the  line  Pc  =  Pc  max  with  the  Pc  versus  fc 
curve  gives  the  minimum  amount  of  control  attention  required,  fc,req'  f°r 
given  system  to  meet  Pc  specifications.  The  difference  between  this  amount  of 
attention,  and  the  total  available  for  the  entire  task  is  the  residual  work¬ 
load  available  for  monitoring 


f 


m, avail 


f 

c  ,req 


(31) 


The  process  of  comparing  the  candidate  display  systems  is  now  clear.  As  an 
example,  one  may  observe  Figure  3.  Clearly  display  system  1  is  superior  to 
display  system  2,  since  less  control  workload  (or  required  control  attention) 
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is  needed  to  meet  the  required  performance  level.  Moreover,  more  attention  is 
available  for  monitoring  duties  when  using  display  system  1. 


Figure  2.  Conceptual  Control  Performance  Figure  3.  Guidelines  for  Evaluating 

Versus  Workload  Curve.  Control/Display  Designs. 

In  the  terrain  following  task  we  compute  the  control  performance  metric  P  , 

Eq.  (20),  for  the  candidate  systems  given  the  constraints  fj  =  0.2,  0.4,  0.6, 

0.8  and  1.0.  Also,  the  pertinent  RMS  flight  path  errors  (erms)  and  g-levels 
(grros)  at  the  same  fT  values  are  obtained.  These  results  are  summarized  in 
Table  2. 

TABLE  2.  PREDICTED  CONTROL  PERFORMANCE  (Pc) , 

FPE,  (ft)  AND  g-STRESS  (g's)  RESULTS 


Display  System^"''*^- 

0.2 

Q.  4 

0.6 

0.8 

1.0 

P 

.84 

.92 

.95 

.96 

.97 

Status 

e 

rms 

28.6 

22.6 

20.4 

19.0 

18.0 

9 

rms 

.54 

.50 

.48 

.47 

.46 

P 

.87 

.95 

.97 

.98 

.99 

Tunnel 

e 

26.2 

20.4 

18.0 

16.6 

15.8 

g 

rms 

.50 

.46 

.44 

.42 

.42 

P 

.93 

.98 

.99 

>.99 

>.99 

Predictor 

e 

rms 

22.4 

17.6 

15.6 
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13.8 

rms 

.44 

.41 

.40 

.38 

.39 

P 

.94 

.99 

>.99 

>.99 

>.99 

Tunnel  +  Predictor 

e 

rms 

21.0 

16.2 

14.6 

13.6 
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g 

.45 

.42 

.41 

.40 

.39 

P 

>.99 

>.99 

>.99 

>.99 

>.99 
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e 

15.0 

12.2 

11.4 

11.0 

10.6 

9 

.41 

.39 

.38 

.37 

.37 

rms 

_ 
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Using  these  numerical  results,  the  display  systems  attain  the  rank  ordering 
as  shown  in  Figure  4.  Using  Eq.  (31)  we  are  now  able  to  compute  the  required 
control-attention,  fCfreq  (workload) ,  and  the  available  monitoring  attention, 
fm, avail'  for  each  of  the  candidate  display  systems.  We  assume  fT~-8-  These 
results  are  summarized  in  Table  3. 


TABLE  3. 

WORKLOAD 

ATTENTION 

AND  MONITORING 

RESULTS 

Display 

System 

f 

c  ,req 

fm, avail 

Status 

CO 

A 

A 

0 

Tunnel 

>.8 

0 

Predictor 

.64 

.16 

Tunnel  + 

Predictor 

.48 

.32 

Flight- 

Director 

<.2 

>.6 

Figure  4.  Display  System  Rank  Ordering  - 
Analytical  Results. 


It  is  evident'  from  these  results  that  some  form  of  synthesized  display  must  be 
considered  in  the  A10  aircraft,  as  the  rudimentary  status  display  yields  an 
unsatisfactory  control  performance.  It  is  also  clear  that  the  best  system,  in 
terms  of  reduced  control  workload,  available  monitoring  attention,  and  lowest 
levels  of  g-stress,  is  the  flight  director  display.  In  addition,  fhe  flight 
director's  Pg-versus-f,;.  curve  is  almost  flat  (i.e.  dPc/dfc^O)  for  fc  e  [0.4, 0.8], 
which  indicates  that  this  design  is  robust  with  respect  to  external  attentional 
demands  that  might  be  placed  on  the  pilot.  The  tunnel,  on  the  other  hand, 
exhibits  supprisingly  poor  results,  as  it  ranks  only  fourth,  after  the  predictor, 
and  does  not  meet  the  performance  criterion. 

The  predictor  display  requires  a  very  high  control  attention,  fc  =  .64,  an<3 
and  the  available  monitoring  attention,  fm? avail'  is  only  .16.  Although  the 
design  specifications  are  met,  such  a  display  system  may  not  be  acceptable,  as 
it  would  be  too  sensitive  to  a  possible  degradation  in  control  attention  capac¬ 
ity.  However,  the  integrated  tunnel/predictor  display  system  gives  a  satis¬ 
factory  performance.  Next  we  validate  the  modeling  results  experimentally. 

EXPERIMENTAL  VALIDATION 


A  primary  objective  of  this  study  was  the  validation  of  the  model-based  display 
design  procedure  described  previously.  The  validation  phase  consisted  of 


618 


fixed-base  man-in-the-loop  simulations  of  the  A10  terrain  following  scenario 
for  the  four  synthetic  displays.  The  experiments,  conducted  largely  indepen¬ 
dently  of  the  analytic  effort,  were  performed  at  the  University  of  Connecticut. 


DISPLAY  FORMAT 


A  precursor  to  the  experimental  phase  is  the  design  of  the  display  format,  i.e. 
the  details  of  the  display  panel  layout.  Clearly,  this  is  largely  an  art,  but 
can  be  guided  by  the  results  of  the  display  element  analysis  with  regard  to 
threshold  values  and  scale  range.  Four  basic,  or  status,  displays  were  used 
in  the  experiments  in  addition  to  the  synthetic  display.  The  basic  displays 
were  the  following. 

1.  Error  Indicator.  This  showed  instantaneous  error  about 
the  nominal  terrain-following  path.  We  used  a  vertical 
scale  of  +50  ft  range  (recall  40  ft  is  the  maximum  design 
error) .  The  distance  between  scale  markings/divisions 
was  chosen  as  5  ft,  which  corresponds  to  a  display  thres¬ 
hold  of  ~2.5  ft.* 

2.  Pitch  Indicator.  A  stylized  aircraft  pitch  indicator 
was  used  to  display  0(t)/q(t).  A  maximum  range  +10° 
was  allowed.  Minimum  scale  marking  was  2.5°. 

3.  g-Meter.  Although  vertical  acceleration  was  not  shown 
to  be  of  significance  as  an  observation,  our  analysis 
assumed  that  RMS  g-level  entered  (subjectively)  in  the 
pilot's  cost  functional.  Since  our  simulation  was  fixed- 
base,  the  only  possible  perception  of  g-level  was  via 
visual  stimulus.  Thus,  the  subjects  were  "aware"  of 
their  commanded  accelerations. 

4.  Radar  Altimeter.  This  display  is  essentially  a  duplication 
of  the  error  information,  i.e.  the  difference  in  altimeter 
reading  from  200  ft  is  the  error.  It  is  included  for  those 
cases  where  the  error  indicator  may  be  off-scale,  |e(t)|>50'. 

In  addition,  any  realistic  display  panel  will  likely  contain 
this  information. 

Figure  5  shows  the  display  panel  layout  that  we  used.  The  center  screen  area 
was  set  aside  for  the  specific  synthetic  displays  to  be  investigated.  The 
display  in  the  lower  right  corner  is  associated  with  a  side  monitoring  task 


We  generally  assume  that 
ing  or  0.05°  visual  arc, 
be  less  than  the  control 
display. 


the  display  threshold  is  half  the  minimum  scale  mark- 
whichever  is  the  larger.  The  display  threshold  should 
indifference  threshold,  y  /8,  for  a  well-designed 
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which  will  not  be  addressed  at  this  time.  The  entire  display  was  presented  to 
the  subject  on  a  VS60  graphics  screen;  the  total  display  size  was  14"  x  12". 


Figure  5.  Basic  Display  Panel  Layout  for  Terrain-Following  Simulation. 

The  basic  display  format  of  Figure  5  was  the  same  for  all  experiments.  The 
only  difference  among  the  cases  studied  was  the  form  of  the  synthetic  display. 
These  are  now  described. 


Predictor  Display 

In  this  display  we  present  the  future  terrain  JI(t+T)  and  extrapolated  aircraft 
position  h(t)+Th(t)  where  T=4  sec.  The  display  format  used  is  shown  in  Figure 
6.  Here  the  cross  represents  the  aircraft  flight  vector.  The  "terrain-box" 
represents  an  80' (H)  x 100' (W)  window  centered  on  the  terrain  path  at  a  dis¬ 
tance  D0=4xU0— 1900'  ahead  of  the  aircraft.  Thus,  if  the  subject  kept  the  cross 
within  the  box,  the  linear  prediction  of  future  error  would  be  <40'.  For  con¬ 
venience,  the  cross  was  fixed  at  the  center  of  the  synthetic  display  area, 
i.e.,  only  the  terrain-box  moved. 

Tunnel  Display 

The  stylized  tunnel  display  that  was  programmed  for  the  experiments  is  shown 
in  Figure  7.  The  "tunnel"  consists  of  five  "windows",  separated  in  distance 
by  500'.  Thus,  with  the  extension  lines,  the  tunnel  presents  the  future  path 
some  2500'  -  3000'  ahead  of  the  aircraft.  The  tunnel  is  centered  on  the  nominal 
terrain  path  and  has  longitudinal  dimensions  +40'  (to  correspond  with  e  )  and 
lateral  dimensions  +50'. 
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The  tunnel  (windows)  are  fixed  in  inertial  space.  Thus,  as  the  aircraft  "flies" 
forward,  the  tunnel  windows  move  towards  the  observer.  When  the  leading  window 
reaches  a  minimal  distance  of  100'  from  the  aircraft  it  disappears,  and  a  "new" 
window  appears  at  the  tunnel's  end.  This  gives  the  illusion  of  continual  for¬ 
ward  motion.  The  perspective  view  of  the  tunnel  is  along  the  aircraft's  flight 
vector,  y(t),  i.e.. ,  the  viewing  axes  are  aircraft  centered  with  the  forward 
z-axis  aligned  with  y(t). 

In  the  present  experiments,  tunnel  variations  occur  only  in  the  longitudinal 
axis.  However,  the  computer  simulation  can  treat  tunnel/terrain  and  aircraft 
motion  in  both  longitudinal  and  lateral  axes.  A  complete  description  of  the 
computer  simulation  and  software  may  be  found  in  [12] . 

Integrated  Tunnel  and  Predictor  Display 

This  display  format  is  essentially  a  combination  of  the  two  previous  displays. 
The  integration  has  been  effected  by  adding  an  additional  "window"  to  the 
tunnel  display  at  a  range  Do=1900'  ahead  of  the  aircraft.  This  window  does  not 
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Figure  6.  Predictor  Display  Symbology.  Figure  7.  Tunnel  Display  Format. 


Figure  8.  Integrated  Tunnel  and 
Predictor  Display. 


Figure  9.  Flight  Director  Display 
Symbology. 
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move  towards  the  observer,  but  the  other  windows  pass  through  it.  The  display 
format  is  shown  in  Figure  8.  In  order  that  the  predictor  window  be  visually 
prominent  it  is  shown  brighter  than  the  other  elements  that  make  up  the  tunnel 
display. 

The  velocity,  or  flight-path  vector  is  superimposed  on  the  tunnel  as  a  cross. 
Since  the  tunnel  view  is  centered  on  this  vector,  the  cross  remains  station¬ 
ary  in.  the  center  of  the  viewing  area.  Of  course,  the  tunnel  (which  is  fixed 
in  inertial  space)  curves  up  or  down  depending  on  the  terrain  and  aircraft 
motion.  We  note  that  if  the  tunnel  view  were  centered  on  the  aircraft  pitch 
angle,  then  the  velocity  vector  projection  would  per  force  be  different. 

Flight  Director  Display 

The  flight  director  signal,  y  (t) ,  as  derived  via  the  methodology  described 
earlier  is  given  by 

yFD(t)  =  -•003n<t)  +  .  1277H  (t)  +  .109511  (t) 

+  . 76a (t)  -  . 476q (t)  -  1.2450(t)  +  .0594e(t)  (32) 

This  has  been  implemented  as 

YFD(t)=Cin(t+T?)''C2n(t)  +  -76a(t)“-476q<t)_1-245e  (t)  +  -0594e(t)  (33) 

where  ci  =  .0745,  cj2  =  .0775,  T  =  1.715  sec.  _In  deriving  Eq.  (33)  we  use  the 
approximation  II  (t  +  T)  =  II  (t)  +  TH  (t)  +  (T2/2)  H  (t)  and  equate  coefficients  with 
Eq.  (32)  .  Note  that  ci  may  be  assumed  equal  to  C2  with  little  or  no  observed 
effects.*  In  Eq.  (33),  II(t+T).  is  the  terrain  path  T  sec.  (or  D=T*Uo=s800  ft) 
ahead  of  the  aircraft. 

The  flight  director  display  is,  presented  to  the  subject  in  the  form  of  a  com¬ 
pensatory  tracking  task, .as  shown  in  Figure  9.  The  cross  in  Figure  9  is  sta¬ 
tionary  with  respect  to  the  viewing  area;  the  signal  that  drives  the  box  is 
given  by  Eq.  (33) .  In  ofder  to  distinguish  this  display  from  the  predictor 
the  box  and  cross  sizing  is  different. 


EXPERIMENTAL  DESIGN 


The  fixed-base  experiments  were  conducted  using  a  PDP  11/60  to  simulate  the 
aircraft  equations,  terrain,  and  to  update  the  displays.  The  display  was  pre¬ 
sented  on  a  VS60  graphics  screen  and  refreshed  30  x  per  second.  Pilot  input 
was  via  a  force  stick  controller.  The  specifics  of  the  simulations  are  given 
in  [6]. 


This  is  convenient  as  only  the  path  different  II  (t+T)  -  II  (t)  is  needed  in  the 
flight  director  signal. 
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Four  subjects  for  the  experiments  were  selected  from  the  Air  Force  ROTC  student 
body  at  the  University  of  Connecticut.  The  display  conditions  were  presented 
to  them  using  a  Latin  square  ordering  to  minimize  any  transition  effects  on 
averaged  performance.  A  data  trial  lasted  130  sec.,  the  last  0.06  x  2048  = 
122.88  sec.  of  which  was  used  as  data.  We  recorded  the  2048  samples  of  con¬ 
trol  input  6  and  error  e  for  each  trial.  In  addition,  we  computed  and  recorded 
RMS  values  of  error,  control,  pitch  and  vertical  acceleration  for  each  run. 
Thus,  we  obtained  a  total  of  N  =  8  x  4  =  32  trials  for  each  display  condition. 

It  should  be  noted  that  none  of  the  subjects  had  flight  experience;  two  had 
some  fixed-base  trainer  experience.  Thus,  it  is  quite  likely  that  the  subjects 
were  (uniformly)  not  expertly  trained  on  the  control  task.  However,  it  is 
quite  likely  that  the  relative  differences  in  performance  for  different  dis¬ 
plays  is  not  strongly  dependent  on  absolute  training  level  in  the  present  task. 


EXPERIMENTAL  RESULTS 


Table  4  gives  the  experimental  results  averaged  across  the  four  subjects.  The 
averages  were  computed  first  for  each  subject  and  then  across  subjects  to  ob¬ 
tain  the  grand  averages.  The  standard  deviations  in  the  experimental  results, 
shown  in  parentheses  in  Table  4,  are  the  averaged  intra-subject  variations  and 
not  the  inter-subject  variations.  Thus,  these  numbers  are  indicative  to  run- 
to-run  variability  that  might  be  associated  with  a  single  ("average")  human.* 


TABLE  4.  AVERAGED  EXPERIMENTAL  RESULTS 


Case 

N 

eRMS 

(ft) 

gRMS_1 

(g) 

1 

32 

22.0 

' (3.4) 

0.44 

(0.068) 

2 

30 

25.7 

(4.5) 

0.53 

(0.078) 

3 

33 

17.8 

(2.2) 

0.44 

(0.084) 

4 

36 

8.95 

(1.9) 

0.36 

(0.054) 

The  results  tabulated  in  Table  4  are  quite  interesting  with  regard  to  the  under¬ 
lying  considerations  in  our  display  design  technique.  First  note  the  rank¬ 
ordering  of  displays  with  respect  to  e^g  performance.  Here  we  see  that  the 
tunnel  display  (case  2)  fares  worst.  The  predictor  display  (case  1)  fares 
slightly  better  than  the  tunnel  alone.  The  combined  tunnel  plus  predictor 
display  (case  3)  results  in  a  meaningful  performance  improvement  over  that  of 
(1)  or  (2) .  The  flight  director  display  (case  4)  yields  significantly  better 
performance  than  any  other  display  condition  —  by  a  factor  of  2.  This  is 
highly  encouraging  validation  of  our  flight  director  design/synthesis  procedure. 

The  rank-ordering  of  the  different  display  configurations  is  in  precise  agree¬ 
ment  with  the  analytical  results  of  the  display  design  methodology.  While  the 


If  intersubject  variability  was  included  in  the  standard  deviations,  the 
values  would  increase  by  20-100%. 
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absolute  levels  of  control  performance  between  experiment  and  model  disagree 
slightly,  relative  performance  levels  agree  well.  This  is  demonstrated  by  com¬ 
paring  model  predictions  (at  a  fixed  fc~. 6)  for  cases  1-3  with  the  experimental 
results.  The  experimental  results  for  cases  1-3  are  consistently  higher 

than  the  model  predictions.  An  explanation  for  this  fact  is  that  the  OCM  assumes 
a  well-trained  subject,  whereas  the  actual  subjects  --  not  being  pilots  —  were 
not  fully  trained  with  respect  to  the  A10  dynamics.  While  it  is  possible  to 
model  this  effect  a  posteriori  in  the  OCM  by  increasing  observation/motor  noise 
and/or  Tfj,  this  was  not  an  objective  of  our  efforts.  On  the  other  hand,  the 
absolute  performance  levels  for  model  and  data  in  case  4  are  in  close  agreement. 
The  reason  for  this  is  that  the  "system"  dynamics  as  perceived  by  the  subject 
are  similar  to  K/s.  These  dynamics  are  trivial  to  learn,  so  that  training 
effects  (after  but  a  few  trials)  are  inconsequential. 


CONCLUSIONS 


An  analytic,  pilot  model-based,  display  design  methodology  has  been  applied  to 
study  workload  and  performance  trade-offs  in  a  high-speed,  terrain-following 
task.  The  methodology  combines  pilot  limitations,  aircraft  dynamics  and  per¬ 
formance  requirements  in  order  to  determine  the  requisite  information  that 
must  be  supplied  to  the  pilot. 

Man -in- the -loop  experiments  that  evaluated  the  performance  of  the  four  candi¬ 
date  display  systems  were  conducted  at  the  University  of  Connecticut.  The 
objective  of  these  experiments  was  to  validate  the  overall  display  design  pro¬ 
cedure,  including  the  flight  director  design  synthesis  process.  Two  positive 
conclusions  resulted  from  this  effort. 

VALIDATION  OF  DISPLAY  SYSTEM  PERFORMANCE 

The  fixed-base  experiments  involved  a  terrain-following  control  task.  (A  second¬ 
ary  monitoring  task  was  also  included  but  is  discussed  elsewhere  [6].) 

The  experimental  relative  rank-ordering  of  the  four  display  systems,  based  on 
control  performance,  was  identical  to  that  predicted  analytically  at  the  dis¬ 
play  element  level.  It  suggests  that  a  large  number  of  potential  display  (or 
control  augmentation)  systems  can  be  evaluated  analytically  at  low  expense, 
and  then  the  most  promising  options  can  serve  as  the  candidates  for  subsequent 
manned  simulation.  The  time  and  cost  savings  of  a  model-based  "front-end”  to 
the  complete  design  process  can  be  substantial. 

The  spread  in  absolute  levels  of  performance  between  the  first  three  display 
systems  and  the  flight  director  display  was  found  to  be  much  greater  in  the 
experiments  than  in  the  model  predictions.  Our  explanation  of  this  result  is 
that  the  model  assumes  a  well-trained  pilot,  whereas  the  subjects  were  not 
well-trained  on  A10  dynamics  and  so  their  performance  was  not  at  the  model- 
predicted  levels.  On  the  other  hand,  the  flight  director  essentially  normal¬ 
izes  out  the  aircraft  dynamics,  rendering  the  control  task  much  simpler  and 
requiring  virtually  no  learning.  Here  model  and  data  absolute  performance 

levels  were  commensurate. 

» 
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VALIDATION  OF  FLIGHT  DIRECTOR  DESIGN  PROCEDURE 


The  analytic  technique  for  flight  director  synthesis  is  included  in  the  display 
methodology  at  the  information  level.  Here,  we  optimally  synthesize  or  aggre¬ 
gate  the  information  states  into  a  single  information  variable  that  could  be 
displayed  to  the  pilot.  The  flight  director  signal  is  designed  to  relate  to 
the  pilot  task  objectives,  i.e.  to  minimize  his  workload  and/or  improve  his 
control  performance,  and  to  satisfy  the  pilot's  desired  goal  of  behaving 
approximately  as  a  gain  and  time-delay.  The  man-in-the-loop  simulations  vali¬ 
dated  the  superiority  of  the  flight  director  display  (over  all  others  considered) 
with  respect  to  control  performance.  The  ability  to  analytically  design  a 
flight  director  that  is  in  harmony  with  pilot  control  and  information  processing 
limitations  is  a  major  feature  of  the  methodology.  In  many  situations  flight 
directors  are  "designed"  via  extensive  simulations  and  tuning  using  a  pilot  in- 
the-loop.  The  analytic  design,  when  used  as  prescribed,  can  shorten  this 
experimental  procedure  to  a  great  extent. 
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SUMMARY 

This  paper  addresses  the  problems  associated  with  developing  an  analytical 
representation  for  the  human  pilot  in  helicopter  visual  flight  task  seg¬ 
ments.  A  two-level  hierarchical  model  structure  with  elements  corresponding 
to  the  autonomous  information  processing  and  control  tasks  and  higher  level 
decision-making  functions  is  proposed.  The  utility  of  this  modeling  framework 
for  understanding  or  interpreting  pilot  response  behavior  is  discussed  with 
reference  to  the  visual  approach  to  a  hover  task.  An  information-theoretic 
approach  for  rank  ordering  the  visual  cues  according  to  information  content  is 
developed,  and  applied  to  the  austere  helipad  scenario. 


1.  INTRODUCTION 

An  important  problem  in  helicopter  mission  task  analysis  is  the  assessment 
or  prediction  of  task  performance  and  concomitant  pilot  workload  in  accomp¬ 
lishing  the  mission  objectives.  Task  performance  measures  are  relatively 
easier  to  quantify  and  may  be  inferred  through  a  human  factors  analysis  of  the 
piloting  task.  On  the  other  hand,  finding  an  analytical  expression  for  the 
degree  of  workload  experienced  by  the  pilot  in  performing  a  given  flight  task 
is  not  so  obvious.  This  situation  persists  in  spite  of  the  voluminous  amount 
of  material  published  on  the  subject  of  workload  [1]  in  the  human  factors 
literature.  As  a  re.sult,  the  prevalent  approach  to  pilot  performance  and 
workload  evaluation  is  empirical  and  is  based  upon  the  interpretation  of  pilot 
opinion  ratings  [2]  which  must  be  obtained  through  extensive  piloted  simula¬ 
tions  and/or  flight  tests.  However,  such  an  approach  is  strictly  experimental 


This  effort  was  performed  under  Contract  No.  DAAK51-81-C-0004  from  the 
Applied  Technology  Laboratory,  USARTL  (AVRADCOM)  For  Eustis,  Virginia. 
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and  provides  little  insight  into  the  identification  of  the  underlying  factors 
influencing  task  performance,  pilot  workload,  and  ultimately  pilot  opinion. 

A  fundamental  axiom  in  pilot-vehicle  systems  analysis  states  that  the 
pilot's  acceptance  of  an  aircraft  for  a  particular  mission  under  a  given  set 
of  operating  conditions  involves  a  tradeoff  between  two  key  factors:  1)  task 
performance  as  perceived  by  the  human  pilot,  and  2)  the  degree  of  workload 
imposed  upon  the  pilot  in  achieving  this  performance  level.  The  pilot  in  a 
realistic  flight  control  task  must  use  the  information  available  from  the 
various  sensory  modalities  to  assess  task  performance  and  generate  the  control 
strategy  appropriate  for  accomplishing  the  desired  task  objectives.  Hence, 
for  a  given  acceptance  level  of  task  performance,  pilot  workload  may  be 
related  to  the  information  processing,  decision-making  and  control  require¬ 
ments  imposed  upon  the  human  pilot  in  satisfactorily  accomplishing  the  task 
objectives  . 

Thus,  an  analytical  methodology  based  upon  the  use  of  an  appropriate 
pilot  model  structure  is  needed.  Existing  pilot  models;  namely,  the  describ¬ 
ing  function  model  [3]  and  the  optimal  control  model  [4],  are  too  structured 
for  describing  pilot  behavior  in  visual  flight  control  tasks  encountered  in 
terrain  flight  missions.  These  models  are  primarily  useful  for  describing 
human  response  during  flight  tasks  under  instrument-meteorological  condi¬ 
tions.  The  unique  features  of  helicopter-visual  flight  task  segments,  in 
particular,  the  availability  of  the  extra-cockpit  visual  scene,  preclude  the 
use  of  existing  pilot  model  based  methodologies.  A  "bottom  up”  or  task  moti¬ 
vated  approach  to  pilot  model  formulation  is  needed. 

In  an  effort  to  remedy  this  situation,  the  U.S.  Army  Aeromechanics  Labor¬ 
atory  at  the  NASA  Ames  Research  Center  has  initiated  a  comprehensive,  long¬ 
term  program  committed  to  the  development  of  a  cohesive  analytical  framework 
for  improved  pilot-vehicle  performance  and  workload  analyses  .  This  study  was 
motivated  by  the  desire  to  develop  an  integrated  quantitative  approach  to  the 
design  of  the  pilot-helicopter  interface,  that  will  allow  full  exploitation  of 
both  pilot  capabilities  and  advanced  technological  developments. 

This  paper  Is  organized  into  four  sections.  Section  2  presents  a  pilot 
model  structure  that  is  suitable  for  describing  pilot-behavior  in  a  wide 
variety  of  flight  tasks.  The  utility  of  the  proposed  model  structure  is 
demonstrated  in  Section  3  and  the  Appendix  by  application  to  a  typical  mission 
segment — namely,  the  visual  approach  to  a  hover  over  a  prescribed  landing 
pad.  Problems  associated  with  formulating  an  analytical  representation  of  the 
pilot's  information  processing,  decision-making  and  control  strategies  are 
discussed.  Conclusions  and  recommendations  for  further  work  are  given  in 
Section  4.  Details  may  be  found  in  Reference  5. 


2.  PILOT  MODEL  FORMJLATION 

The  helicopter  pilot  involved  in  a  civilian  or  military  mission  is 
required  to  perform  a  wide  range  of  tasks.  The  specific  nature  of  these  tasks 
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is  dictated  by  a  number  of  factors,  the  most  important  among  these  being:  (1) 
the  mission  objectives,  (2)  the  operating  environment  and  scenario,  (3)  the 
helicopter  type,  configuration  and  dynamic  characteristics,  and  (4)  the  soph¬ 
istication  of  the  groundbased  and  airborne  navigation  and  guidance  system. 

The  principal  objective  of  this  paper  is  to  present  a  conceptual  frame¬ 
work  for  analyzing  human  information  processing,  decision  making  and  control 
behavior  in  flight  segments  representative  of  typical  civilian  and  military 
helicopter  missions.  Iherefore,  this  study  is  limited  to  an  investigation  of 
key  piloting  task  segments  such  as  approach  to  a  hover,  hover  (or  station¬ 
keeping)  and  nap-of-the-earth  (NOE)  using  extra  cockpit  visual  cues  as  the 
primary  source  of  navigation  information.  Hie  pilot  model  structure  formu¬ 
lated  here  is  based  upon  the  premise  that,  although  the  various  missions  and 
flight  segments  appear  to  be  different  at  the  outset,  there  exists  a  sub¬ 
stantial  degree  of  commonality  in  the  nature  of  the  task  functions  and 
requirements  imposed  on  the  pilot/crew. 

Fundamentally,  the  process  of  flying  consists  of  performing  a  hierarchy 
of  tasks  corresponding  to  increasing  levels  of  cognitive  involvement.  As  a 
minimum,  piloting  tasks  may  be  classified  into  two  hierarchical  categories: 

(1)  lower  level  autonomous  information  processing  and  control  tasks,  and  (2) 
higher  level  decision-making  tasks.  A  hierarchical  model  structure  for  the 
pilot  based  upon  the  two  level  task  decomposition  is  shown  in  Figure  1. 

The  autonomous  information  processing  and  control  task  consists  of  a 
sequence  of  four  operations;  a)  cue  selection,  b)  cue  information  processing, 
c)  performance  evaluation,  and  d)  control  law  implementation.  The  pilot  has 
access  to  a  variety  of  cues  from  different  sensory  modalities— namely ,  visual, 
vestibular  (motion),  proprioceptive  and  accoustic.  This  discussion,  however, 
is  limited  to  a  consideration  of  the  visual  cues  as  the  primary  sensory  infor¬ 
mation  available  to  the  pilot.  Generalization  of  the  concepts  to  an  array  of 
sensory  cues  appears  feasible  and  may  be  carried  out  in  subsequent  studies. 

The  extra  cockpit  visual  scene  provides  the  pilot  with  a  rich  array  of 
visual  cues  which  contain  information  about  the  aircraft  situation  with 
respect  to  the  outside  world.  Since  the  pilot  does  not  have  unlimited  data 
processing  capability,  he  must  choose  the  necessary  cues  containing  the  most 
information  content  about  the  aircraft  situation  for  estimating  the  state  of 
the  aircraft.  The  cue  selection  and  information  processing  elements  perform 
the  navigation  task  of  determining  the  aircraft  position  and  orientation  from 
the  available  visual  cues. 

The  performance  evaluation  element  compares  the  estimated  aircraft  state 
with  some  desired  reference  and  determines  whether  or  not  a  control  response 
is  warranted.  A  simple  deadzone  nonlinearity  represents  one  plausible  form 
for  the  performance  evaluation  logic  element.  In  this  case,  control  action  is 
dictated  only  when  the  estimated  aircraft  state  deviates  from  the  desired 
value  beyond  some  prescribed  threshold  level.  The  control  law  element  per¬ 
forms  the  task  of  implementing  control  actions  that  tend  to  drive  the  aircraft 
state  to  the  desired  reference  situation. 
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Figure  1.  Hierarchical  Pilot  Model  Structure 


The  decision  center  performs  the  higher  level  cognitive  functions  con¬ 
sisting  of  a)  internal  model  formulation,  b)  task  organization,  c)  performance 
definition  and  assessment,  and  d)  structure  adaptation. 

The  term  "internal  model”  refers  to  the  form  of  the  pilot's  perception  of 
his  own  (aircraft)  situation  with  respect  to  the  outside  world.  This  is 
equivalent  to  making  assumptions  about  a  frame  of  reference  that  may  be  used 
by  a  pilot  in  performing  the  flying  tasks.  The  standard  approach  is  based 
upon  physical  considerations  and  assumes  that  the  pilot  adopts  an  earth-fixed 
(cartesian,  spherical  or  cylindrical)  reference  frame  with  it's  origin  at  some 
meaningful  point  on  the  earth  (e.g.,  helipad  center).  This  approach  leads  to 
the  definition  of  the  perceived  aircraft  situation  in  terms  of  the  familiar 
position  (x,y,z)  and  orientation  ( ^ , 6 , 4>)  state  vector  representation.  How¬ 
ever,  it  is  quite  likely  that  a  pilot  may  use  an  entirely  different  precept 
based  upon  phenomenological  rather  than  physical  considerations.  An  example, 
is  the  use  of  geometric  features  in  the  visual  scene  as  internal  states  for 
accomplishing  a  given  flight  task. 

The  process  of  task  organization  consists  of  defining  the  particular 
task,  as  perceived  by  the  pilot,  into  an  analytical  framework  that  is  consis¬ 
tent  with  the  pilot's  internal  model  or  representation  of  the  flight 
scenario.  The  mission  objectives  must  be  translated  into  a  desired  mission 
profile  consisting  of  a  concatenation  of  the  individual  phases  and  task  seg¬ 
ments.  Eventually,  this  must  lead  to  a  definition  of  an  acceptable  or  desired 
performance  envelope  for  the  instantaneous  aircraft  state  as  a  function  of 
some  independent  variable  such  as  time  or  range-to-go.  In  an  earth  referenced 
internal  frame  of  reference,  this  usually  implies  the  specification  of  a 
desired,  range  dependent,  reference  flight  trajectory  and  speed  profile  for 
the  aircraft. 

The  function  of  the  structure  adaptation  element  in  the  decision  center 
is  to  choose  the  structure  and  parameters  for  the  cue  selection  matrix,  infor¬ 
mation  processing  algorithm,  the  performance  evaluation  logic  and  the  control 
strategy  algorithm  that  are  most  suited  for  the  particular  task  segment  and 
operating  scenario.  This  function  reflects  the  well-known  capability  of  human 
pilots  to  fly  intelligently  by  adapting  their  information  processing  and 
control  strategies  to  varying  internal  or  external  conditions. 

The  proposed  model  structure  is  extremely  versatile  and  allows  for  con¬ 
siderable  flexibility  in  selecting  the  mathematical  representation  for  each  of 
the  individual  elements.  In  the  absence  of  certain  knowledge  or  conviction 
regarding  the  actual  operations  performed  by  a  pilot,  the  only  rational 
recourse  is  to  hypothesize  plausible  mechanisms  to  represent  the  various 
piloting  processes . 

The  utility  of  this  hierarchical  model  structure  for  understanding  and 
interpreting  pilot  response  behavior  in  realistic  flight  control  tasks  is 
illustrated  in  the  next  section  with  reference  to  the  visual  approach  to  a 
hover  task. 
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3.  VISUAL  APPROACH  TO  A  HOVER  TASK 


The  ability  of  the  helicopter  pilot  to  fly  a  visual  approach  to  hover  over 
a  prescribed  landing  zone  is  a  well-known  and  routinely  accepted  skill.  What 
is  not  understood,  however,  is  the  process  by  which  the  human  pilot  is  able  to 
perform  this  difficult  task.  Based  upon  the  pilot  model  structure  formulated 
in  the  previous  section,  the  human  pilot  may  be  described  as  performing  the 
following  three  functional  tasks: 

1)  Acquire  and  process  the  necessary  visual  cues  to  estimate  the  posi¬ 
tion  and  orientation  of  the  aircraft  with  respect  to  the  prescribed 
landing  pad, 

2)  Assess  the  estimated  aircraft  situation  or  state  by  comparing  it  with 
the  desired  objectives,  and  decide  whether  a  corrective  control 
action  is  warranted,  and 

3)  Apply  a  corrective  control  action  so  as  to  bring  the  aircraft  to  an 
acceptable  or  desired  state. 

This  section  presents  the  plausible  rationale  and  schemes  that  may  be  used 
by  a  pilot  in  accomplishing  the  above  three  functional  tasks. 

Section  3.1  describes  the  processes  that  may  be  involved  in  the  pilot's 
selection  of  the  visual  cues  inherent  in  the  visual  scene.  An  estimation- 
theoretic  rationale  for  cue  selection  that  is  based  upon  quantifying  and  rank¬ 
ordering  the  information  content  in  the  visual  cues  is  proposed  and  described 
In  the  Appendix.  The  usefulness  of  this  approach  is  demonstrated  by  applica¬ 
tion  to  a  generic  cu6  set  that  Is  representative  of  the  helipad  approach 
scenario. 

Having  discussed  an  approach  for  selecting  the  visual  cues,  the  next  step 
is  to  describe  the  pilot's  use  of  such  cues  during  the  helicopter  visual 
approach  and  hover  tasks.  Section  3.2,  discusses  this  issue  from  the  pheno¬ 
menological  as  well  as  physical  viewpoints.  Problems  associated  with  inter¬ 
preting  the  pilot's  information  processing,  decision-making  and  control 
functions  into  a  mathematical  closed  loop  simulation  of  the  visual  approach  to 
a  hover  task  are  discussed. 

Finally,  Section  3.3,  presents  some  thoughts  on  factors  contributing  to 
pilot  workload  within  the  proposed  pilot  modeling  framework. 


3.1  Selection  of  the  Visual  Cues 

The  extra-cockpit  visual  scene  provides  the  pilot  with  a  wide  variety  of 
cues  which  he  may  use  for  aircraft  navigation.  Just  what  these  cues  are,  and 
how  they  may  be  used  by  the  human  pilots  are  questions  that  have  been  a  sub- 
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ject  of  intense  research  and  debate  among  scientists  over  several  decades. 
There  are  at  least  two  distinct  approaches  towards  determining  the  answers  to 
these  questions — the  physical  and  the  phenomenological.  The  physical  approach 
is  based  upon  treating  the  visual  scene  as  an  image  (provided  by  the  retina  of 
the  eye)  which  contains  information  about  the  aircraft  state  relative  to  the 
earth.  Information  theoretic  methods  can  then  be  used  to  determine  and  rank 
order  the  information  content  in  the  visual  cues.  In  contrast,  the  phenomeno¬ 
logical  approach  starts  with  the  notion  that  the  human  pilot  alone  is  best 
qualified  to  identify  visual  cues  in  the  extra  cockpit  scene  and  define  ways 
to  use  them.  Neither  of  these  two  approaches  is  without  drawbacks.  The 
physical  approach,  by  definition,  depends  entirely  on  a  deductive  method  and 
may  lead  to  conjectures  that  appear  entirely  speculative  and  unreasonable  from 
the  human  information  processing  viewpoint.  The  phenomenological  approach 
which  depends  upon  the  use  of  pilot  questionnaires  is  criticized  for  just  the 
opposite  reasons — namely,  the  over-emphasis  on  pilot  interpretation  of  the 
visual  cues  which  presumes  that  a  pilot  is  consciously  aware  of  what  he  truly 
does,  and  furthermore,  that  he  can  articulate  this  knowledge  in  specific 
terms.  Therefore,  both  of  the  above  two  approaches  were  followed  in  this 
study.  A  pilot  questionnaire  designed  to  elicit  answers  to  key  questions  was 
prepared.  Results  obtained  from  this  questionnaire  study  will  be  reported 
separately.  The  discussion  in  this  paper  is  limited  to  an  analysis  of  the 
visual  cues  from  an  estimation/information-theoretic  viewpoint . 

The  literature  on  visual  cues  is  extensive  [6-10]  .  However,  it  is  gener¬ 
ally  agreed  that  monocular  cues  are  more  than  adequate  for  visual  piloting 
tasks  [10]  .  The  monocular  cues  commonly  listed  [11]  are:  (1)  linear  perspec¬ 
tive,  (2)  relative  motion  or  movement  parallax,  (3)  apparent  versus  real  size 
and  shape,  (4)  interposition,  (5)  relative  contrast  and  brightness,  (6)  aerial 
perspective,  (7)  texture,  and  (8)  accommodation. 

Examination  of  the  above  list  of  cues  indicates  that  the  majority  of  the 
cues  are  geometric  in  nature  and  can  be  described  using  methods  of  perspective 
geometry.  The  geometric  visual  cues  available  to  the  pilot  during  an  approach 
to  a  helipad  are  discussed  next. 


3.1.1  Geometric  Visual  Cues 


The  visual  scene  as  sensed  by  the  retina  is  a  two-dimensioned  (2-D)  image 
of  the  three-dimensional  (3-D)  world.  As  such,  this  2-D  image  is  ambiguous 
and  does  not  provide  a  unique  characterization  of  the  actual  3-D  visual 
scene.  This  is  because  the  projection  or  mapping  of  a  3-D  object  onto  a  plane 
is  many-to-one.  The  ability  of  humans  to  resolve  this  ambiguity  by  associat¬ 
ing  a  unique  external  3-D  representation  is  based  upon  prior  knowledge,  learn¬ 
ing,  and  interpretation  by  the  central  nervous  system.  A  perspective  static 
view  of  the  approach  to  a  hover  task  scenario  is  shown  in  Figure  2.  This 
discussion  is  limited  to  helicopter  motion  in  the  vertical-longitudinal 
axes.  The  coordinates  (x,0,h)  represent  the  instantaneous  position  of  the 


633 


position  of  the  helicopter  in  a  helipad  reference  frame  with  its  origin  at  the 
center  of  the  pad.  Figure  3  shows  the  static  forward  view  out  of  the  cockpit 
as  seen  by  the  pilot.  For  the  purpose  of  this  discussion,  the  pilot’s  line- 
of-sight  is  assumed  to  be  level  and  aligned  with  the  helipad's  longitudinal 
axis.  Thus,  for  a  helipad  ABEC  of  length  2L  and  width  2D,  the  projected 
forward  image  on  a  canopy  plane  placed  at  a  normal  distance  c  from  the  pilot's 


HeJlpad  M  (x^.y^.O) 


Figure  2.  Helipad  Approach  Scenario 


eye  corresponds  to  the  trapezoid  A'B'E'C'.  Similarly,  any  horizontal  line 
element  PQ  on  the  helipad  maps  into  a  corresponding  line  element  P'Q'  in  the 
projected  image.  The  three  variables  &  ,  w  and  n  shown  in  Figure  3 

X0  X0 


Figure  3.  Perceived  Extra-Cockpit  Visual  Scene 


characterize  the  projected  image  of  the  helipad.  For  the  scenario  described 
by  Figure  2  and  3,  J l  ,  w^  and  n  can  be  shown  to  be  nonlinear  functions  of 
the  aircraft  coordinates  xOand  h.  Thus, 
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The  static  perspective  view  of  the  helipad  is  a  source  of  a  number  of 
visual  cues  to  the  pilot.  Plausible  candidates  include  (1)  perceived  length 
of  the  helipad,  (2)  perceived  widths  of  the  helipad  at  the  far  end,  near  end 
and  center,  (3)  perceived  inclination  of  the  helipad  edges  to  the  horizontal, 
and  (4)  perceived  depressions  of  the  far  end,  near  end  and  center  of  the 
helipad  with  respect  to  the  horizon.  The  cues  defined  above  only  represent 
one  plausible  set.  Other  reasonable  static  cues  involving  ratios  or  sums  of 
perceived  lengths  and  angles  may  quite  well  be  used  by  the  human  pilot . 
Furthermore,  the  framework  of  perspective  geometry  may  also  be  used  to  formu¬ 
late  visual  motion  cues  [12]  (i.e.,  motion  cues  obtained  from  the  dynamic 
f rame-to-f rame  changes  in  the  perceived  visual  scene)  and  texture  cues. 

Based  upon  the  above  discussion,  the  pilot  has  a  multitude  of  visual  cues 
available  to  him  for  performing  the  navigation,  guidance  and  control 
functions.  For  the  scenario  described  in  Figures  2  and  3,  only  two  indepen¬ 
dent  measurements  or  cues  are  necessary  for  estimating  instantaneous 
helicopter  longitudinal  range  x  and  altitude  h.  However,  the  pilot  is  not 
limited  to  using  only  two  cues.  Just  how  the  pilot  chooses  a  given  set  from 
the  vast  number  of  plausible  cues  is  not  yet  understood.  The  approach  taken 
here  is  to  assume  that  the  pilot  chooses  the  visual  cues  which  contain  and 
provide  the  most  amount  of  information  for  the  navigation  task— namely,  the 
task  of  estimating  vehicle  position  and  orientation  with  respect  to  the  land¬ 
ing  pad.  The  Appendix  presents  an  analytical  method  for  determining  and 
ranking  the  information  content  in  the  visual  cues.  The  discussion  is  limited 
to  geometric  visual  cues  in  the  visual  approach  to  a  hover  task;  however, 
extension  to  any  other  set  of  cues  is  straightforward.  An  information  theo¬ 
retic  analysis  of  the  visual  cues  defined  by  Eq  .  (1)~(3)  above  shows  that  the 
amount  of  information  contained  in  these  cues  grows  with  increasing  values  of 
xq  and  D,  or  equivalently  an  enlarging  f ield-of-view  in  the  elevation  and 
azimuth  planes,  respectively. 


3.2  Task  Utilization  of  the  Visual  Clues 

The  previous  section  and  the  Appendix  describe  a  plausible  rationale  that 
a  pilot  may  use  in  selecting  a  set  of  visual  cues  from  the  extra-cockpit 
scene.  The  material  in  this  section  attempts  a  similar  formalization  of  the 
functional  process  involved  in  the  pilot's  utilization  of  the  visual  cues  in 
the  approach  to  a  hover  task.  A  first  step  in  this  process  is  to  try  and 
understnd  the  pilot's  subjective  interpretation  or  perception  of  the  guidance 
requirements  for  the  given  task  as  described  next. 
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3.2.1  Guidance  Commands 


A  reasonable  way  to  start  is  by  defining  the  task  objective.  For  a  visual 
approach  to  a  hover  task  the  principal  objective  is  to  bring  the  helicopter 
from  some  initial  cruise  altitude  and  velocity  to  a  hover  over  the  prescribed 
landing  pad.  In  order  to  accomplish  this  goal,  the  pilot  must  apply  the 
necessary  control  inputs  (primarily  longitudinal  cyclic  and  collective  for  the 
straight-in  profile)  to  the  helicopter  to  fly  a  desired  descending  and  decel¬ 
erating  approach  profile.  This  implies  that  the  pilot  has  developed  some 
explicit  and/or  implicit  notion  of  a  desired  approach  profile  or  corridor 
during  the  approach  to  a  hover  task. 


The  existence  of  such  preferred  visual  approach  profiles,  or  corridors, 
is  clearly  demonstrated  by  the  results  of  a  flight  experiment,  by  Msen,  et  al. 
[13] ,  showing  definite  patterns  or  characteristic  shapes  in  the  altitude, 
ground-speed,  and  deceleration  profiles  of  visual  approaches  for  heli¬ 
copters.  TWo  key  features  are  apparent:  (1)  the  altitude  versus  range 
profiles  resemble  a  typical  three  segment  approach  trajectory  consisting  of  a) 
a  constant  altitude  segment,  b)  a  constant  flight  path  angle  (6.5°  ~  12.5°) 

approach  and  c)  a  final  constant  altitude  segment  prior  to  hover,  and  (2)  the 
ground  speed  versus  range  profiles  correspond  to  a  constant  g  (  »  .03g)  decel¬ 
eration,  except  for  the  last  1000  ~  1500  feet  prior  to  hover  when  the  speed 

is  suddenly  reduced  by  an  almost  exponential  increase  in  deceleration . 
Furthermore,  definite  trends  in  these  patterns  with  initial  altitude  and  speed 
are  observed.  These  results  reinforce  the  belief  that  a  trained  pilot 
deliberately  chooses  to  fly  a  desired  approach  profile  under  a  given  set  of 
operating  and  environmental  conditions . 


Just  how  the  pilot  performs  the  guidance  task  is  a  matter  of  conjecture. 
Formulation  of  a  plausible  guidance  strategy  depends  upon  what  one  assumes  for 
the  pilot's  internal  model  or  representation  of  the  outside  world.  The 
approach  taken  here  is  based  upon  the  assumption  that  the  pilot  perceives  the 
outside  environment  in  terms  of  the  helicopter  position  and  orientation  with 
respect  to  a  helipad  centered  reference  frame  (i.e.,  cartesian,  cylindrical, 
or  spherical  coordinates)  .  Thus  for  a  straight-in  approach  (along  the  x  axis 
of  the  helipad  as  shown  in  Figure  2),  the  helicopter  state  or  position  coor¬ 
dinates  may  be  represented  as: 


x 


(x,h)T 


in  cartesian  coordinates. 


or 


(4) 


x 

where  Y 


(Y,  h)  or 

„  -1  h 

tan  — 


(x,Y)T 


in  cylindrical  coordinates 


(5) 

(6) 


is  the  flight  path  angle.  Therefore  the  desired  altitude  and  ground  speed 
profiles  may  be  expressed  as: 
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:  Altitude  Command 


(7) 


h  =  h  (x) ,  or 
c  c  * 

Yc  =  Yc(x) 


and 


v  H  x  =  v  (x) 
c  c  c 


Speed  Command 


(8) 


Note  that  the  formulation  of  the  guidance  commands  given  by  Eq  .  (4 ) — ( 8 ) ,  is 
motivated  by  physical  concepts  commonly  used  in  autopilot  design.  However,  it 
is  likely  that  a  pilot  may  use  an  entirely  different  precept  based  upon  pheno¬ 
menological  rather  than  physical  considerations.  An  example,  is  the  use  of 
feature  patterns  in  the  visual  scene  as  internal  states  for  acconplishing  a 
given  flight  task.  The  key  lies  (1)  in  recognizing  visual  cues  or  feature 
patterns  which  exhibit  invariance  properties  for  a  desired  flight  profile,  and 
(2)  in  using  these  identified  cues  for  structuring  the  guidance  commands. 

There  have  been  several  studies  in  the  past  [14-23] ,  directed  towards  the 
subject  of  visual  cues  during  the  approach  and  landing  task.  These  studies 
show  that  the  pilot  may  be  using  a  number  of  invariance  patterns  in  the  visual 
scene.  Examples  for  a  constant  glide  slope  visual  approach  are  (1)  the 
constant  depression  of  the  intended  touchdown  point  below  the  horizon,  and  (2) 
the  stationarity  of  the  aim  point  (focus  of  expansion)  with  respect  to  other 
objects  in  the  visual  scene.  Thus,  for  a  constant  desired  flight  path  angle 
Yc  ,  the  guidance  command 

=  constant  (e.g.,  6.5°  ~  12.5°)  (9) 


may  be  equivalently  described  ss 


Si.  -  c  tan  Y  =  c  -  =  constant  (10) 

0  c  x 

where  is  the  depression  of  the  center  of  the  helipad  (intended  aim  point) 

below  the  horizon.  Note  that  in  the  absence  of  a  horizon,  an  alternate  invar¬ 
iance  pattern  for  constant  Yc  is 
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where  w^,  i  and  are  defined  by  Eq  .  (l)-(3)  .  Equation  (11)  states  that  a 
constant  gliaeslope  approach  to  the  center  of  the  helipad  can  be  flown  by 
maintaining  the  ratio  of  the  perceived  distance  of  the  helipad  center  from  the 
near  end  of  the  helipad  to  the  perceived  near  end  width  at  a  constant  value. 


Equations'  (10)-(11)  show  that  a  constant  glide  slope  approach  can  be  flown 
without  using  cues  that  permit  an  explicit  estimation  of  the  ground  range  x  or 
altitude  h.  However,  these  constant  glideslope  conditions  only  hold  for  a 
portion  or  segment  of  the  overall  approach  trajectory.  Therefore,  the  heli¬ 
copter  pilot  must  use  additional  cues  that  provide  information  for  range  and 
altitude  estimation. 
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The  above  discussion  indicates  that  several  hypotheses  can  be  advanced  to 
describe  the  guidance  command  structures  that  may  be  adopted  by  the  humam 
pilot.  The  following  paragraphs  discuss  the  issues  involved  in  integration  of 
such  guidance  schemes  with  the  navigation,  performance  assessment,  and  control 
functions  of  the  helicopter  pilot  in  the  visual  approach  to  a  hover  task. 


3 .2 .2  Closed-Loop  Simulation  of  the  Visual  Approach  to  a  Hover  Task 

Before  one  can  proceed  with  a  closed  loop  simulation  of  the  visual 
approach  to  a  hover  task,  it  is  necessary  to  have  a  detailed  mathematical 
representation  for  each  of  the  elements  in  the  pilot  model  structure.  The 
preceding  material  presented  some  plausible  schemes  for  selecting  the  visual 
cues  and  for  defining  the  guidance  laws.  This  section  discusses  the  formu¬ 
lation  of  algorithms  for  the  cue  processing  (i.e.,  state  estimation),  perfor¬ 
mance  evaluation  and  control  elements  in  the  hierarchical  pilot  model 
structure . 

As  stated  earlier,  the  pilot  model  structure  of  Figure  1  is  extremely 
versatile  and  allows  for  considerable  flexibility  in  choosing  the  form  of  the 
individual  functional  elements.  Thus,  the  describing  function  [3]  and  optimal 
control  model  [4]  structures  represent  two  specific  formalizations  within  the 
hierarchical  model  structure.  The  describing  function  lumps  all  aspects  of 
human  information  processing  and  control  into  a  "black  box”  input-output 
representation.  At  best,  higher  level  cognitive  functions  are  accounted  for 
by  allowing  for  an  adaptation  of  the ■ describing  function  parameters  (gain, 
lead,  lag,  time  delay,  etc.)  with  changes  in  the  operating  flight  condi¬ 
tions.  In  contrast,  the  optimal  control  model  formulation  represents  an 
attempt  towards  developing  a  model  structure  that  is  homomorphic  to  known 
psycho-physical  aspects  of  human  response  behavior.  Specifically,  the  model 
consists  of  three  elements  in  tandem — a  Kalman  filter  or  state  estimator, 
an  optimal  predictor,  and  an  optimal  linear  quadratic  controller.  The  optimal 
control  model  structure  is  normative;  that  is,  the  model  describes  or  predicts 
what  the  human  should  do  as  opposed  to  what  he  actually  does  do.  Furthermore, 
according  to  the  optimal  control  model,  the  structures  for  the  pilot's  cue 
selection,  information  processing,  performance  evaluation  and  control  are 
predefined  within  the  linear,  quadratic,  and  Gaussian  (LQG) ,  estimation  and 
control  theoretic  framework.  However,  even  a  cursory  review  of  the  visual 
approach  to  a  hover  task  is  sufficient  to  indicate  that  such  a  model  in  its 
standard  form  may  not  be  suitable,  however  attractive  its  use  may  appear,  for 
describing  the  human  pilot  in  this  situation. 

During  visual  meterological  conditions  (VMC),  the  out-of-the-window  visual 
scene  and  the  standard  cockpit  instruments  are  considered  to  be  the  two  pri¬ 
mary  sources  of  information  available  to  the  pilot.  Hence,  it  is  reasonable 
to  assume  that  the  quality  of  the  available  cues  in  the  visual  scene  will  have 
a  direct  impact  on  the  pilot's  selection  of  the  visual  cues  (including  those 
from  cockpit  instruments)  and  their  subsequent  utilization  in  the  navigation 
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(i.e.,  state  estimation),  guidance  (i.e.,  desired  approach  profiles),  perfor¬ 
mance  evaluation,  and  control  tasks.  Consequently,  the  shape  of  the  desired 
altitude  and  velocity  guidance  profiles  would  be  expected  to  adapt  to  the 
given  operational  scenario  and  environmental  conditions.  In  this  context,  the 
nominal  visual  approach  profiles  for  altitude  and  ground  speed  versus  range, 
[Figures  5  and  8,  respectively,  in  reference  13],  were  obtained  at  NASA 
Wallops  Flight  Center  under  normal  weather  conditions  using  helicopters  equip¬ 
ped  with  standard  cockpit  instruments.  It  is  important  to  note  that  the 
nominal  flight  profiles  are  not  likely  to  remain  unchanged  under  austere 
environmental  conditions  or  in  situations  where  the  pilot  is  denied  infor¬ 
mation  from  the  cockpit  instruments. 

The  helipad  landing  scenario  described  by  Figure  2  is  definitely  austere, 
and  was  deliberately  chosen  to  permit  an  understanding  of  the  visual  cues  from 
first  principles.  Unfortunately,  there  is  practically  no  data  available  for 
this  scenario  that  could  be  used  for  objectively  defining  the  guidance  pro¬ 
files  and  the  corresponding  navigation  and  control  algorithms  adopted  by  the 
human  pilot.  In  the  absence  of  data,  the  only  option  available  is  to  proceed 
with  the  simulation  using  plausible  mathematical  representations  for  the 
various  modules  in  the  human  pilot  model.  A  matrix  of  off-line  computer 
simulation  tests  may  be  defined  corresponding  to  varying  degrees  of  informa¬ 
tion  processing  and  control  law  sophistication.  Thus,  for  example,  the 
navigation  algorithm  for  estimating  the  aircraft  state  from  the  given  visual 
cues  can  be  as  simple  as  a  deterministic  inversion  of  the  nonlinear  mapping 
describing  the  visual  cues  or  involve  the  implementation  of  sophisticated 
model-based  filters  (i.e.,  a-6  filter,  complementary  filter  of  Kalman  filter 

in  that  order)  .  Similarly,  the  performance  evaluation  and  control  module 
structures  can  range  from  the  classical  or  optimal  (LQG)  feed-forward/feed¬ 
back  control  laws  to  an  alternative  control  concept,  recently  introduced  by 
Rault  and  Richalet  [24,25],  termed  "model  predictive  heuristic  control"  [24], 
"scenario  predictive  control"  [25],  "model  algorithmic  control"  [26],  or 
"output  predictive  algorithmic  control"  [27]  . 

The  alternate  control  technique  is  radically  different  in  concept  from  the 
existing  methods  (i.e.,  classical  control,  LQG  control,  etc.)  which  rely  on 
the  explicit  feedback  of  the  instantaneous  state  estimate  to  achieve  the 
desired  objectives.  Instead,  the  technique  is  based  upon  the  computation  of 
the  control  input,  every  At  seconds  (not  necessarily  constant) ,  so  as  to 
minimize  the  predicted  error  over  some  future  "horizon  of  prediction"  between 
the  actual  and  clesired  output  profiles  (e.g.,  altitude  and  range-rate 
profiles)  .  This  control  technique  has  been  successfully  applied  to  a  number 
of  practical  aircraft  problems  [26-27]  .  This  approach  appears  to  provide  the 
most  natural  framework  for  describing  human  control  strategy  in  situations 
such  as  the  visual  approach  to  a  hover  task. 

* 

The  above  discussion  points  out  the  wide  range  of  plausible  information 
processing  and  control  algorithms,  hence,  the  enormous  dimensionality  of  the 
resulting  closed-loop  simulation  test  matrix  that  must  be  investigated. 
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3.3  Interpretation  of  Pilot  Workload 


Pilot  workload  is  usually  defined  as  an  aggregate  scalar  measure  of  the 
degree  of  effort  or  difficulty  experienced  by  the  pilot  in  accomplishing  a 
task  at  a  given  level  of  performance.  However,  the  difficulty  experienced  by 
a  pilot  depends  upon  the  nature  and  complexity  of  the  functional  requirements 
Imposed  upon  the  pilot  in  performing  the  task.  Therefore,  it  is  preferable  to 
conceive  of  workload  as  a  vector  metric  with  elements  representing  the  contri¬ 
butions  of  the  pilot's  information  processing,  decision-making  and  control 
tasks.  Then,  aggregate  workload  can  be  defined  as  some  linear  or  nonlinear 
function  of  the  vector  components.  Three  approaches  for  quantifying  pilot 
workload  have  been  presented  in  the  literature.  They  are:  (1)  subjective 
workload  or  demand  ratings,  (2)  secondary  task  performance  measures  of  spare 
mental  capacity,  and  (3)  physiological  indicators  such  as  galvanic  skin 
resistance  (GSR),  pupil  dilation,  and  heart  rate.  A  comprehensive  survey  of 
the  workload  literature,  including  a  useful  taxonony  of  the  workload  measures 
and  assessment  methods,  may  be  found  in  reference  [1].  These  techniques, 
however,  do  not  represent  valid  analytical  measures  for  pilot  workload;  there¬ 
fore,  methods  based  upon  an  analytical  representation  or  model  of  the  human 
pilot's  information  processing,  decision-making  and  control  strategies  have 
been  proposed.  Thus,  the  describing  function  model  based  workload  measure  is 
defined  to  be  some  nonlinear  function  of  the  model  parameters  associated  with 
task  difficulty — namely,  the  lead  parameters  (time  constants  fo  the  first 
order  zeros)  reflecting  prediction  requirements  and  the  time  delays  signifying 
task  urgency  [28]  .  Similarly,  the  optimal  control  model  based  measure  of 
workload  is  assumed  to  be  proportional  to  the  fraction  of  total  attention 
capacity  that  must  be  devoted  to  the  task  in  order  to  maintain  a  given  level 
of  performance  [29]  .  Clearly,  the  validity  of  such  measures  is  only  as  good 
as  the  veracity  of  the  pilot  model  structures  on  which  they  depend.  As 
discussed  earlier,  the  applicability  of  the  existing  pilot  models  to  the 
visual  approach  to  a  hover  task  is  questionable,  and  a  "bottom-up"  or  task 
motivated  approach  to  pilot  modeling  and  performance/workload  assessment  is 
recommended . 

Analytical  measure  reflecting  the  degree  of  effort  of  difficulty  exper¬ 
ienced  by  the  pilot  performing  the  task  are  required.  In  the  visual  approach 
to  a  hover,  the  visual  cues  provided  by  the  visual  scene  and  standard  cockpit 
Instruments  are  the  pilot's  primary  source  of  position  and  orientation  infor¬ 
mation.  Therefore,  pilot  workload  would  be  expected  to  increase  with 
degradation  in  the  quality  (i.e.,  information  content)  of  the  visual  scene. 
Pilot  workload  for  a  given  scenario  and  acceptable  level  of  task  performance 
must  depend  upon  (1)  the  complexity  or  sophistication  of  the  information 
processing,  decision-making  and  control  algorithms,  and  (2)  the  amount  of 
control  effort  required.  The  control  effort  required  can  be  quantified  easily 
in  terms  of  its  variance  and  frequency  content  (i.e.,  spectral  density  char¬ 
acteristics).  However,  defining  analytical  measures  for  information 
processing,  decision  making,  and  control  law  complexity,  is  not  so  obvious. 

This  is  especially  true  when  dealing  with  human  cognitive  functions  and 
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abilities  as  opposed  to  computer  capabilities.  A  case  in  point  is  the  pheno¬ 
menal  human  capability  for  effortlessly  analyzing  and  processing  natural  cues 
as  provided  by  the  human  senses.  Here  the  task  of  spatial  orientation  and 
performance  assessment  becomes  easier  as  the  quality  of  the  visual  scene 
improves,  and  vice-versa.  This  observation  is  counter-intuitive  from  the 
computer  processing  viewpoint.  The  key  lies  in  recognizing  that  the  human 
comes  equipped  with  a  built-in  cognitive  capability  that  allows  the  parallel 
processing  of  natural  cues.  Hence,  information  processing  complexity,  as 
perceived  by  the  pilot,  decreases  with  increasing  number  of  visual  cues  as 
long  as  these  cues  are  integrated  and  conform  to  natural  (i.e.,  compatible 
with  human  sensory  apparatus)  format. 

Unfortunately,  making  similar  observations  and  comments  about  the  pilot's 
control  strategy  is  not  so  straight-forward.  For  example,  consider  two 
control  strategies  based  upon  classical  (output  feedback)  and  linear  quadratic 
control  which  are  designed  to  yield  the  same  performance  and  require  identical 
amounts  of  control  effort.  Clearly,  from  a  computer  implementation  viewpoint, 
the  linear  quadratic  controller  is  much  more  complex  than  a  classical 
controller  since  the  former  requires  a  solution  to  a  matrix  Riccati  equation 
for  determining  the  control  gains.  This  observation  also  holds  from  the 
piloting  viewpoint  unless  one  assumes  that  the  human  also  comes  equipped  with 
a  built-in  processor  for  solving  Riccati  equations.  Obviously  more  experimen¬ 
tal  work  aimed  towards  identifying  pilot's  preferences  in  control  strategy 
selection  is  needed. 

The  above  discussion  points  out  the  difficulties  involved  in  formulating 
analytical  measures  for  pilot  workload.  However,  much  understanding  about  the 
piloting  task  requirements  can  be  gained  by  performing  an  analysis  of  the 
tradeoff  between  the  information  content  in  the  visual  cues  and  required 
control  effort,  for  a  given  level  of  task  performance. 


4.  CONCLUSIONS  AND  RECOMMENDATIONS 

A  systematic  pilot  model-based  framework  for  analyzing  and  interpreting 
pilot  response  behavior  in  visual  flight  tasks  is  proposed.  A  generic  pilot 
model  structure  incorporating  the  known  human  functions  of  information 
acquisition/processing,  decision  making  and  control  is  formulated.  The 
utility  of  this  model-based  approach  towards  understanding  the  piloting  task 
is  discussed  with  references  to  the  helicopter  visual  approach  to  a  hover 
scenario. 

The  extra-cockpit  visual  scene  Is  assumed  to  be  the  pilot's  primary  source 
of  information.  The  helicopter  pilot  must  use  this  information  to  perform  the 
required  navigation,  guidance  and  control  tasks.  Therefore,  the  pilot  work¬ 
load  in  accomplishing  the  task  at  a  satisfactory  performance  level  must  be 
intimately  dependent  upon  the  quality  of  the  visual  scene  from  the  information 
viewpoint.  To  understand  this  relationship,  it  is  first  necessary  to  have 
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some  quantitative  measures  of  information  content  in  visual  cues.  Hence,  an 
estimation-theoretic  approach  towards  quantifying  and  rank-ordering  the  infor¬ 
mation  contained  in  the  visual  cues  available  from  the  extra-cockpit  scene  is 
described.  The  basic  concepts  are  elucidated  by  analyzing  and  ranking  the 
geometric  visual  cues  to  the  pilot  during  an  approach  to  a  helipad. 

The  weakest  link  in  applying  a  model-based  approach  lies  in  not  being  able 
to  define  what  a  pilot  actually  does  with  the  information  provided  by  the 
visual  cues.  In  other  words,  no  firm  data  base  of  rationale  exists  for 
selecting  specific  algorithms  to  describe  the  information  processing, 
decision,  and  control  elements  in  the  proposed  model  structure.  However, 
plausible  mechanisms  for  describing  the  pilot’s  utilization  of  the  visual  cues 
in  performing  the  flying  task  are  discussed. 

Pilot  workload  is  defined  as  a  vector  quantity  with  elements  corresponding 
to  the  information  processing,  decision-making  and  control  components.  An 
interpretation  of  workload  in  terms  of  the  required  model  complexity  and 
control  effort  for  accomplishing  a  task  is  presented. 

In  summary,  a  cohesive  structure  for  the  analysis  and  interpretation  of 
human  pilot  behavior  in  realistic  flight  scenarios  has  been  developed. 

However,  an  experimental  data-base  is  needed  to  develop  analytical  formula¬ 
tions  for  the  pilot's  information  processing,  decision-making  and  control 
algorithms.  Two  types  of  piloted  simulation  experiments  are  recommended: 

1)  Laboratory  fixed  base  simulation  studies,  and  2)  Flight  test  experiments. 
The  purpose  of  the  labortory  tests  would  be  to  determine  some  fundamental 
characteristics  of  human  perception  in  the  context  of  visual  flight  tasks  . 
Flight  tests,  on  the  other  hand,  are  needed  to  provide  a  realistic  data  base 
for  interpreting  the  effects  of  the  visual  scene  content  on  the  piloting  task. 


APPENDIX 

Information  Theoretic  Analysis  of  the  Visual  Cues 

The  purpose  of  this  Appendix  is  to  demonstrate  the  utility  of  estimation- 
theoretic  concepts  for  determining  the  information  content  in  the  visual  cues 
provided  by  the  extra-cockpit  scene.  The  basic  concepts  of  estimation  and 
information  are  introduced  first  in  generic  terms.  This  is  followed  by  an 
example  which  ranks  the  specific  visual  cues,  defined  by  Eq  .  (l)-(3),  for  the 
visual  approach  to  .a  hover  tank. 

The  visual  cues  described  by  Eq  .  (1 )— ( 3)  are  nonlinear  functions  of  the 
aircraft  state  with  respect  to  the  helipad.  Formally,  these  cues  may  be 
expressed  as  a  nonlinear  mapping 

z  =  f  (x)  (A.l) 
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T 

where  x.  =  (xl>x2>  *  *  xn)  n  dimensional  aircraft  state  vector, 

T 

-  (z^,Z2,  •  .  zp)  is  the  p-dimensional  visual  cue  vector, 
and 

T 

f  =  (f^,f2,  •  •  fp)  are  P  nonlinear  functions  that  map  the, 

aircraft  state  vector  x_ into  the  p  individual  visual  cues  z^ 

However,  the  human  pilot  is  limited  in  his  ability  to  detect  small 
changes  in  the  visual  cues  z^  .  Therefore,  a  visual  threshold  must  be 
introduced  in  defining  the  perceived  visual  cues.  Typically,  two  types  of 
visual  thresholds  are  considered  -  (1)  a  resolution  or  detection  threshold, 
and  (2)  a  discrimination  threshold.  A  resolution  threshold  refers  to  the 
human  ability  to  detect  small  changes  in  a  variable  from  some  explicit 
reference.  The  discrimination  threshold  describes  the  ability  of  the  human  to 
distinguish  a  small  change  in  a  variable  in  the  absence  of  an  explicit 
reference  condition.  The  visual  approach  to  a  hover  task  must  be  performed  in 
the  absence  of  any  explicit  reference  or  desired  condition.  Therefore, 
discrimination  threshold  dominates  the  resolution  threshold  and  must  be 
considered  in  analyzing  the  perceptual  process.  The  discrimination  threshold 
in  perceiving  a  visual  cue  z^  may  be  determined  according  to  the  Weber-Fechner 
law  [30]  as 

6ziT  =  %  '  zi  (A*2) 

where  ■=  — —  (A. 3) 


Furthermore,  a  threshold  can  equivalently  be  treated  as  an  additive 
observation  or  perceptual  noise  v  with  a  variance 


(A  .4) 


Equation  (A. 3)  is  based  upon  the  results  of  an  experiment  conducted  by 
Wewerinke  [31-App.A]  where  vz  is  treated  as  an  observation  error  (or  noise  in 
perceiving  z^.  Thus,  ^ 


z  =  z  +  v  (A. 5) 

-p  -  -z 


where  _z^  is  the  equivalent  perceived  visual  cue  vector  and  v^  is  the 
equivalent  observation  noise  vector  with  covariance  matrix 


E  (^z  -z)  =  R 


0 


0 


R 


2 


**P 


(A  .6) 
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Eqs.  (A.5)-(A.6)  define  the  perceived  cues  available  to  the  human 
pilot.  The  pilot's  information  processing  task  is  to  determine  the  best 
possible  estimate  of  the  aircraft  state  x_ from  the  noisy  perceived  cues  . 
Let  the  pilot's  apriori  (before  perceiving  the  cue)  estimate  of  x.  he  x  having 
an  uncertainty  or  error  defined  by  the  covariance  matrix 


E  [(x  -  x)  (x  -  x)T]  =  M  (A  .7) 

Then,  a  reasonable  estimate  x^  that  may  be  obtained  by  the  pilot  is  the 
weighted-least-squares  estimate  which  minimizes  a  quadratic  cost  [32] 


J  =  |  [  (x  -  x)T  M  *  (x  -  x)  +  -  Hx)T  R  ^  -  Hx)](A.8) 


This  estimate  (which  is  identical  to  the  minimum  variance  or  maximum  likeli¬ 
hood  estimate  for  gaussian  assumptions)  is  given  by  the  standard  Kalman  filter 
equation 


x  =  x  +  P  HT  R-1  (zp  -  f (x ) ) 


(A  .9) 


where 


x=x 


(A  .10) 


is  the  linearized  measurement  matrix  at  x  *  x,the  apriori  state  estimate,  and 


P  =  E  [  (x  -  x)  (x  -  x)T]  (A. 11) 

is  the  estimation  error  covariance  or  dispersion  matrix.  Furthermore,  it  can 
be  shown  that 

-1  -1  T  -1 

P  *  M  +  H  R  H  (A  .1 2) 

Since  HTR-*H  is  at  least  positive  semi -definite  (i.e.,  >  0) ,  Eq .  (A. 12)  shows 

that  the  estimation  error  covariance  P  after  using  the  perceived  cues  _z^  is 
never  larger  than  M,  the  estimation  error  covariance  before  using  the  cues. 
Thus’,  the  use  of  the  perceived  cues,  on  the  average,  reduces  the  uncertainty 
(i.e.,  error  covariance)  in  the  knowledge  of  the  true  state  x_.  Equivalently, 
the  effect  of  cue  utilization  is  to  increase  the  amount  of  information 
available  to  the  pilot  about  the  instantaneous  aircraft  state.  This 
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interpretation  is  based  on  recognizing  that  the  inverse  of  the  error 
covariance  or  uncertainty  is  information.  Thus  and  P-*  may  be  considered 
as  the  apriori  (i.e.,  before  cue  utilization)  and  aposteriori  (after  cue 
utilization)  information  matrices,  respectively.  Thus,  in  the  information 
context,  the  term  (H^R  ^H)  in  Eq .  (A.12)  reflects  the  additional  information 
about  the  state  provided  by  the  perceived  cue  vector  z  .  If  the  apriori 
information  about  the  vehicle  state  x_  is  poor,  then  =  0  and  the 
information  about  the  state  after  cue  processing  is 

I  =  D*  =  (HTR_1H)  .  (A  .13) 

It  should  also  be  noted  that  the  dispersion  matrix  D  is  the  error  covariance 
matrix  after  measurement  (assuming  no  apriori  information)  of  the  visual 
cues.  Furthermore,  D  can  be  given  a  geometric  interpretation  in  terms  of 
constant  likelihood  (or  uncertainty)  hyperellipsoids 

(x  -  x)T  D  (x  -  x)  =  £2  (A  .14) 

where  £  is  a  constant. 

The  probability  or  likelihood  that  the  true  aircraft  state  x^ lies  within 
a  hyperellipsoid  (A  .14)  depends  upon  the  specific  values  for  n  and  £,  and  can 
be  calculated.  Thus,  for  a  two  state  vector  (n=2),  the  probability  of  finding 
the  true  x_ inside  the  £= 1  ellipse  is  0.394,  inside  1=2  ellipse  is  0.865,  and 
inside  £=3  ellipse  is  0.989.  These  ellipsoids  for  £=1,2,3  are  usually 
referred  to  as  39,  86  and  99  percent  likelihood  ellipsoids. 

The  size  and  shape  of  the  likelihood  ellipsoid  reflects  the  amount  of 
information  about  the  aircraft  state  contained  in  the  perceived  cue  vector. 
Therefore,  measures  based  upon  the  Information  matrix  I  or  equivalently,  the 
dispersion  matrix  D  ,  may  be  used  for  quantifying  and  ranking  the  amount  of 
information  contained  in  a  given  set  of  visual  cues.  Typically,  the  following 
scalar  measures  are  used: 

*  n 

J.  =  Det  D  =  II  X. 

j  _ i 


*  “ 

J,  =  Tr  D  =  EX 
i=l 

J3  =  Max  {Xi(D*)} 

i  =  1 ,2,  . .  ,  n 


(A  .15) 


(A  .16) 


(A  .17) 
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where  A^,  i  =  1,2,  ,  n  are  the  eigenvalues  of  D  .  Note  that  the  square 

roots  of  the  eigenvalues  /  A  correspond  to  the  magnitudes  of  the  principal 
axes  of  the  hyperellipsoid  along  their  respective  eigenvector  directions.  The 
determinant  of  D  ,  (J^),  is  a  measure  of  volume  of  the  one  sigma(£=l) 
likelihood  ellipsoid.  Trace  D  ,  (J2),  gives  a  measure  of  the  average  mean- 
squared  estimation  error,  and  the  maximum  eigenvalue  of  D  ,  (J3),  represents 
the  largest  principal  semi-axis  of  the  hyperellipsoid  and  hence  the  worst  case 
estimation  error  variance. 


The  ranking  of  the  visual  cues  based  upon  the  dispersion  matrix 
implicitly  takes  into  consideration  the  effects  of  cue  sensitivities  and 
thresholds.  This  is  apparent  from  the  definition  of  the  dispersion  matrix  as 


*  A  T  -1 
D  =  ITR  H 


3f 

where  H  =  is  the  sensitivity  matrix  and  R  is  a  diagonal  observation 


noise  matrix  with  diagonal  elements  reflecting  the  discrimination  threshold 
levels  for  each  of  the  available  cues . 


A .1  Information  Ranking  of  the  Visual  Cues  -  An  Example. 

An  analytical  framework  for  determining  and  ranking  information  content 
in  the  visual  cues  has  been  presented.  Application  of  this  methodology  for 
evaluating  the  visual  cues  during  the  approach  to  a  hover  task  is  discussed 
next.  The  visual  cues  available  to  the  pilot  are  limited  to  the  geometric 
cues  described  by  Eq  .  (l)-(3),  which  are  repeated  here  for  convenience 


l 

xo 

c  h 

x  -  x0 

(1) 

w 

c  D 

(2) 

xo 

x  -  x0 

n 

=  tan  * 

h 

D 

(3) 

If  the  rectangular  helipad  shown  in  Figure  2  is  the  only  source  of  visual  cues 
to  the  pilot,  then  a  finite  set  of  cues  corresponding  to  the  perceived 
lengths,  widths  and  inclinations  may  be  used  by  the  pilot  for  the  state 
estimation  task.  However,  for  most  realistic  situations,  the  pilot  has  access 
to  far  more  visual  cues  than  those  provided  by  a  helipad  alone.  Thus,  useful 
cues  may  be  available  from  a  much  larger  field-of-view  than  that  covered  by 
the  helipad  image.  Conceptually,  a  larger  field-of-view  in  azimuth  and 
elevation  for  the  visual  scene  may  be  represented  by  superimposing  a  grid 
structure  on  the  ground  plane  with  lines  parallel  to  the  longitudinal  and 
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lateral  edges  of  the  helipad.  Mathematically ,  this  is  equivalent  to  allowing 
a  range  of  discrete  values  for  Xq  and  D  in  Eq .  ( 1)— (3)  for  the  visual  cues. 
Figure  A  .1  shows  the  elevation  (side  view)  and  plan  (to'p-down)  views  of  the 
helipad  approach  scenario.  The  pilot’s  view  outside  the  window  is  limited  in 
elevation  (6)  and  azimuth  (3)  by  virtue  of  the  canopy  configuration. 


Side  View 


Top  View 


Figure  A  .1 .  Field-of-View  Limitations 


For  given  6  and  3,  the  values  of  xQ  and  D  must  be  bounded: 


and 


„  h 

xn  <  x  -  - - T 

u  tan  6 

D  <  2  (x  -  x0)  tan  | 


(A  .18) 


where  (x,h)  are  the  longitudinal  range  and  altitude  of  the  helicopter  with 
respect  to  the  helipad  center. 

For  a  given  Xq  and  D,  Eq  .  (l)-(3)  represent  three  visual  cues  that  may  be 
used  by  the  pilot  for  estimating  his  ground  range  x  and  altitude  h.  At  least 
two  cues  are  needed  to  estimate  x  and  h.  Thus,  the  pilot  may  select  any  of 
the  following  four  sets  of  cues: 
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versus  xn  Figure  A-3  .A  (D  )  versus 


for  a  finite  or  continuum  of  values  of  i,  that  may  correspond  to  horizontal 
grid  line  elements  of  varying  length  .  Hie  use  of  additional  extra-cockpit 
visual  cues  provides  more  information  to  estimate  the  helicopter  state  and 
should  result  in  reduced  estimation  error  covariance  and  thus  improved  state 
estimates.  Just  how  many  visual  cues  are  sufficient  would  depend  upon  the 
nature  of  the  curve  describing  the  trade-off  between  the  number  of  cues 
(abscissae)  and  the  cumulative  information  (ordinate)  .  Usually,  the  number  of 
cues  selected  is  truncated  on  the  knee  of  the  cumulative  information  versus 
the  number  of  cues  curve  where  the  amount  of  incremental  information  provided 
by  adding  a  cue  is  not  justified  by  the  cost  or  effort  of  acquiring  and 
processing  that  cue. 
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1,  Introduction.  This  paper  presents  highly  intriguing  preliminary 


results  concerning  the  mathematical  model  of  human  pilots  hovering  over 
a  moving  landing  pad  in  an  X-22A  VTOL  aircraft.  The  results  are  in  the 
form  of  a  time  domain  mathematical  model  which  was  determined  from  transient 
responses  of  the  X-22A.  The  pilot,  using  both  a  head-up-display  for  land¬ 
ing  pad  position  information  and  the  outside  visual  scene  for  attitude 
information,  was  asked  to  precisely  hover  over  a  simulated  landing  pad 
which  would  move  in  semi-random  step-like  jumps.  Since  the  X-22A  (with 
its  translation  rate  control  system)  was  also  driven  by  simulated  ship 
wake  turbulence  and  natural  turbulence,  the  pilot's  task  was  to  track 
step  inputs  while  correcting  for  random  disturbances. 

Unlike  many  other  research  efforts  the  research  reported  here  is 
based  upon  time  domain  parameter  identification  methods.  The  advantage 
of  the  time  domain  approach  is  that  it  effectively  duplicates  the  situa¬ 
tion  observed  in  flight,  i.e.  the  need  to  make  discrete  maneuvers  while 
minimizing  disturbances.  The  assumption  of  stationarity  is  never  made 
and  for  good  reason  since  very  few  piloting  tasks  are  even  remotely 
stationary  or  time  invariant  in  nature. 

The  proposed  pilot  model  encompasses  two  primary  control  policies 
which  were  observed  in  the  flight  data.  The  first  is  a  bang-bang  control 
policy  for  amplitude  limited  gross  maneuvers  while  the  second  policy  is 
an  error  minimization/disturbance  rejection  linear  control  policy.  Efforts 
are  under  way  to  identify  the  unknown  parameters  of  the  model  from  the 
X-22A  flight  data  using  state  and  parameter  estimation  techniques.  Identi¬ 
fied  pilot  models  for  various  X-22A  dynamic  configurations  will  then  be 


correlated  v/ith  k? 


ir.le  handling  qua!  i  ties'  in  order  to  formulate  a 


control  system  design  procedure  directly  incorporating  handling  qualities 

considerations. 
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2.  Flight  Task  Description.  Evolving  Naval  tactics  are  today  forc¬ 
ing  naval  helicopters  airborne  in  all  weather  and  on  ever  smaller  ships. 

The  flight  data  upon  which  this  research  is  based  was  generated  in  an  attempt  to 
study  the  problem  of  visual  landing  of  a  helicoDter  or  VTOL  on  a  small  non¬ 
aviation  ship  in  bad  Weather.  Funded  through  the  Naval  Air  Development 
Center,  the  Cal  span  Advanced  Technology  Center  devised  a  flight  experiment 
(References  1,  2)  using  the  X-22A  variable  stability  aircraft,  Figure  1, 
in  which  the  pilot  was  instructed  to  remain  positioned  over  a  moving  land¬ 
ing  pad  in  the  presence  of  artifically  generated  ship  wake  turbulence  and 
some  unavoidable  natural  atmospheric  turbulence.  Since  the  task  was  a 
visual  one,  the  pilot  could  observe  absolute  position,  velocity,  and 
altitude  by  viewing  the  outside  world  through  the  cockpit  canopy.  The 
position  of  the  landing  pad  was  presented  through  a  head-up-display  (HUD) 
using  a  display  format  shown  in  Figure  2.  The  information  presented  was 
symbolic  since  the  field  of  view  of  the  HUD  effectively  precluded  a 
pictorial  presentation  of  the  landing  pad  and  the  ship  superstructure. 

Salient  features  of  the  display  are  the  round  fixed  aircraft  symbol 
(scaled  to  a  10  foot  diameter)  and  the  altitude  ladder  with  rung  separa¬ 
tion  scaled  to  10  feet.  The  longitudinal  and  lateral  displacement  from 
the  landing  pad  (square  symbol)  are  presented  in  plan  view  in  a  heading 
axis  system.  Referring  to  Figure  2,  closure  with  the  pad  would  require 
forward  and  right  stick.  Height  above  the  landing  pad  is  depicted  by  the 
separation  of  the  double  dumbell  and  the  aircraft  symbol,  in  effect  an 
elevation  view  of  the  vertical  situation.  As  in  the  x-y  situation  presen- 
t.i i. Ion ,  i he  I i  ol  is  in  the  fly  to  sense,  that  is,  * he  rhanbell  symbol 
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The  pursuit  tracking  task  presented  to  the  pilot  via  the  HUD  involv¬ 
ed  discrete  semi-random  step-like  jumps  in  landing  pad  position.  In  the 
first  portion  of  any  given  pilot  evaluation  the  pad  position  changed  once 
every  twenty  or  thirty  seconds  in  either  x  position  separately,  y  separate 
ly  or  both  x  and  y  positions  simultaneously.  Since  the  vertical  axis 
flight  control  system  employed  altitude  hold  the  pilot's  task  of  moving 
to  the  new  landing  pad  and  then  hovering  accurately  over  the  pad  in  the 
presence  of  disturbances  was  overall  a  two  axis  task  with  definable 
portions  of  one  axis  activity.  In  the  second  portion  of  any  pilot  evalu¬ 
ation  the  task  involved  the  same  sequence  of  step -like  landing  pad  move¬ 
ments  applied  in  the  Y  and  Z  pad  positions.  This  portion  of  the  pilot's 
evaluation  was  therefore  a  three  axis  task  since  X  stationkeeping  remain¬ 
ed  a  demanding  chore. 

After  a  step  change  in  landing  pad  position  the  pilot  performed  a 
gross  maneuver,  examples  of  which  are  discussed  later,  in  which  the 
pilots  objective  was  to  rapidly  reposition  over  the  new  pad  position. 

Once  repositioning  over  the  pad  was  carried  out  the  pilot  attempted  to 
hover  over  the  pad  to  within  an  accuracy  of  one-half  symbol  width 
(+  5  feet). 

3.  Flight  Control  System.  The  test  vehicle,  the  X-22A  operated  by  the 
Calspan  Advanced  Technology  Center  under  sponsorship  from  the  Naval  Air 
Development  Center,  was  configured  so  that  the  response  to  a  steady  pitch 
or  roll  stick  input  generated  a  steady  state  inertial  translational  velo¬ 
city  in  the  x  or  y  direction  respectively.  This  translational  rate 
command  (TRC)  system, also  called  a  velocity  command  control  system, was 


a  step  control  stick  command  the  response  of  vehicle  translational  velo¬ 
city  and  attitude  is  shown  in  Figure  4.  Note  that  the  X-22A  must  change 
attitude  in  order  to  tilt  the  thrust  vector  in  order  to  translate  either 
side-to-side  or  fore  and  aft. 

The  TRC  system  was  mechanized  using  feedback  of  inertial  velocity 

(«’,  K*)»  attitude  (K.,  K)  and  angular  rates  (K„,  K  ) .  The  resulting 
x  y  e  <P  h  P 

closed  loop  transfer  functions  for  translational  velocity  (x  or  y)  and 
attitude  (e  or  <j>)  are  given  below. 


-KeM  g 
e 


(s-Xu)(s^(KqMie-Mq)s+KeHSe)  +  9(MU«X-M5e) 


(l+S/X)(l  +  (2?/a)n)s+(s/un)2) 


0) 


6ES  9  (l+s/x)(l+(2c/a>n)s+(s/a3n)2) 


The  system  gains  were  selected  so  that  the  dynamic  response  and 
stick  force  system  characteristics  were  identical  in  both  pitch  and  roll 
axes. 

The  yaw  axis  flight  control  system  was  not  a  significant  contributor 
to  the  flight  experiment  and  the  pilot  was  able  to  keep  his  feet  off  the 
rudder  petals.  While  the  altitude  hold  vertical  axis  flight  control 
system  was  of  significance  in  the  Calspan  flight  test  program  it  is  not 
of  consequence  for  this  paper  since  we  deal  here  with  only  X  and  Y  maneu¬ 
vering  at  constant  altitude. 
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4.  The  Data  Base.  The  primary  purpose  of  the  Calspan/NADC  flight  experi¬ 
ment  was  to  determine  the  effect  of  variations  in  comnand  path  gain  (steady 
state  translational  velocity  per  inch  of  stick  deflection,  Kp,  and 

path  mode  time  constant  (a  in  Equations  1)  upon  vehicle  handling  qualit¬ 
ies.  In  order  to  explore  these  factors  the  attitude  and  angular  rate 
feedback  gains  were  adjusted  oh  the  majority  of  the  flight  configurations 
so  as  to  maintain  an  inner  loop  attitude  system  with  natural  frequency  of 
2.5  radians  per  second  and  damping  ratio  of  .8.  The  velocity  feedback 
gains  and  forward  loop  gains  were  varied  to  realize  path  mode  time  cons¬ 
tants  ranging  from  1.5  to  4  seconds  and  command  path  gains  from  3  to  12 
ft/sec/in.  The  inherent  flexibility  of  the  X-22A's  variable  stability 
control  system  made  this  task  possible  through  a  bank  of  pilot  adjustable 
potentiometers  located  in  the  cockpit. 

Approximately  86  piloted  evaluations  of  the  type  described  in  the 
last  section  have  been  obtained  by  the  authors  from  the  Naval  Air  Develop¬ 
ment  Center  on  fourteen  magnetic  tapes.  Each  tape  contains  time  histories 
of  all  signals  pertinent  to  the  experiment. 

Following,  each  evaluation  the  pilots  (there  were  two  pilots  in  the 
available  data  base)  were  instructed  to  provide  Cooper-Harper  pilot  rat¬ 
ings  and  to  provide  detailed  comnentary.  This  data  is  available  in 
Reference  2. 

5.  Research  Objectives.  The  objectives  of  this  study  are  listed  below: 

i  To  formulate  a  useful  mathematical  model  of  the  human  pilot  while 
performing  a  discrete  tracking  task  and  station  keeping 
duties  in  a  hovering  VTOL  (X-22A). 
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11  To  employ  time  domain  parameter  Identification  methods  In 


order  to  determine  the  numerical  values  of  all  unknown  terms 
In  the  pilot  model.  This  step  is  to  be  repeated  for  a 
broad  range  X-22A  dynamics  in  the  class  of  translational 
rate  command  systems. 

iii  To  determine  useful  measures  of  closed  loop  pilot/vehicle 
performance. 

iv  to  correlate  identified  pilot  models  and  closed  loop  vehicle 
performance  measures  with  known  handling  qualities. 

v  To  generalize  the  analysis  so  as  to  develop  a  control  system 
design  tool  capable  of  predicting  closed  loop  pilot/vehicle 
handling  qualities. 


6.  Research  Results..  The  results  to  date  include  the  formulation  of  a 
time  domain  nonlinear  mathematical  model  of  the  human  pilot  based  upon 
a  detailed  and  continuing  analysis  of  the  X-22A  flight  data.  This  model 
is  presented  and  justified  in  the  remainder  of  this  section.  Objectives 
ii-  v  remain  to  be  satisfied  in  this  continuing  research  program. 

6.1  Typical  Flight  Data.  Figures  5  through  8  contain  typical  flight 
data  showing  the  pilot/vehicle  response  to  a  step  change  in  x-longitudinal 
landing  pad  position.  Each  figure  contains  three  time  history  plots.  The 
top  one  labeled  X  ERROR  shows  the  error  between  the  x-pad  position  and 
the  X-22A  position.  The  steplike  jump  in  X  ERROR  is  the  result  of  the 
sudden  landing  pad  position  change  described  in  Section  2.  The  center 
plot,  labeled  DE,  is  the  fore-and-aft  position  of  pilot's  stick,  i.e.  the 


pilots  primary  output  in  this  x-positioning  task. 


The  lower  plot 
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labeled  Q,  is  the  resulting  pitch  rate  of  the  X-22A.  Table  1  shows 
the  path  mode  time  constant  A,  the  command  path  gain  K^,  and  the  Cooper- 
Harper  pilot  rating  for  the  data  presented  in  the  Figures. 


Table  1 

Figures 

Configuration 

x(sec) 

K^(ft/sec/in) 

Cooper-Harper  PR 

5,  6 

208A 

2.75 

5.5 

6 

7,  8 

206B 

2.0 

6.0 

4 

Careful  examination  of  these  time  histories  show  that  following  the 
step  landing  pad  change  the  pilot  moves  the  pitch  controller  with  two 
types  of  policies.  The  first  stick  motion  policy  involves  bang-bang  type  of 
motion  in  which  the  pitch  control  is  moved  to  a  pilot  perceived  control 
limit,  held  there  for  typically  5  seconds,  moved  to  a  limit  in  the  oppo¬ 
site  direction,  held  there  for  another  5  seconds  and  then  returned  to 
zero.  This  strategy  is  certainly  based  upon  the  pilots  desire  to  move 
as  rapidly  as  possible  to  the  new  landing  pad  position  and  stop  over  the 
pad  without  overshoot.  The  fact  that  the  control  used  is  of  limited  mag¬ 
nitude  should  be  of  no  surprise  if  it  is  recalled  that  in  a  tilt-to- 
translate  vehicle  such  as  the  X-22A  there  are  large  pitch  attitude  excur¬ 
sion  associated  with  translation  and  it  seems  reasonable  that  the  pilot 
will  tolerate  these  only  up  to  some  maximum  values. 

The  second  policy  apparent  in  the  pilot's  stick  consists  of  smaller 
amplitude  oscillations  at  frequencies  less  than  1  Hertz.  Examination 
of  the  pitch  rate,  Q,  plot  shows  strong  correlation  between  the  high 
frequency  stick  and  pitch  rate  activity.  This  type  of  pitch  stick 
activity  is  probably  due  ta  a  combination  of  pilots  inability  to  accu¬ 
rately  carry  out  his  desired  control  policy  (i.e.  motor  noise)  and  the 
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pilots  desire  to  surpress  the  X-22A's  response  to  natural  turbulence 
and  synthetically  generated  shipwake  turbulence. 

In  summary  two  types  of  pilot  strategies  can  be  hypothesized  from 
Figures  5-8.  The  first  is  a  gross  acquisition  maneuvering  characterized 
by  large  amplitude  bang-bang  control  motion.  The  second  is  a  lower 
amplitude  higher' frequency  disturbance  rejection.  The  pilot  model  pro¬ 
posed  subsequently  has  been  designed  to  deal  with  both  strategies. 

6.2  Bang-Bang  Optimal  Control.  For  the  germane  problem  of  minimizing 
the  time  to  reposition  over  the  landing  pad  subject  to  amplitude  limited 
control  the  theory  of  optimal  control  leads  to  a  rather  elegant  bang-bang 
optimal  control  policy.  The  previous  section  and  Figures  5-8  strongly 
suggest  exactly  this  policy  was  employed  by  the  X-22A  pilot. 

If  the  second  order  portion  of  equations  which  characterize  primar¬ 
ily  the  attitude  response  characteristics  is  ignored  the  resulting  dif¬ 
ferential  equations  describing  x-position  and  translational  velocity 
are  of  second  order.  With  this  reduced  order  model  the  optimal  control 
can  be  computed  to  track  a  step  change  in  pad  position  in  minimum  time 
with  control  limited  to  plus  or  minus  one  inch.  The  optimal  bang-bang 
control  computed  in  this  way  is  shown  in  Figure  9  which  is  most  directly 
comparable  to  the  in-flight  measured  situation  shown  in  Figure  6.  Com¬ 
parison  of  these  two  figures  shows  reasonably  accurate  agreement  on 
switching  times  and  gives  strong  evidence  towards  establishing  the 
veracity  of  the  optimal  bang-bang  pilot  control  policy  postulated  earlier. 

The  next  sub  section  proposes  a  model  structure  which  integrates 
the  two  pilot  policies  which  have  just  been  postulated. 


660 


6.3  Proposed  Pilot  Model.  A  pilot  model  is  described  in  this  section 
which  includes  the  two  strategies  hypothesized  in  previous  sections.  The 
model  shown  in  Figures  10-12  is  based  upon  the  following  assumptions. 

1.  The  pilot;  upon  seeing  a  step  change  in  pad  position,  xre^  will 
compute  a  bang-bang  control  policy  6  ,  so  as  to  reposition 
over  the  pad  in  minimum  time.  The  maximum  allowable  positive 
and  negative  limits  on  control  activity  (6^,  of  Figure  12) 
are  not  constrained  to  be  the  same  allowing  for  a  possible 
pilot  preference  to  translate  faster  forward  with  nose  down 
and  the  ground  in  sight. 

2.  Once  the  bang-bang  control  policy  is  formulated  the  pilot 
determines  mentally  the  desired  aircraft  motion  i.e.  the 
motion  the  aircraft  would  follow  as  a  result  of  his  bang- 
bang  control  policy.  This  becomes  the  desired  aircraft 
trajectory  (XD(t ) ) . 

3.  Because  the  pilot  is  unable  to  perfectly  carry  out  his 
desired  control  policy  there  will  be  errors,  motor  noise 
m(t),  in  the  actual  positioning  of  his  control. 

4.  The  pilot's  output  is  delayed  because  of  perceptual  and 
neuromuscular  pure  time  delays  and  his  actual  movement  is 
assumed  to  be  modeled  as  a  first  order  lag.  The  latter 
seems  necessary  because  of  the  obviously  finite  slopes  of 
the  pilots  output,  DE,  at  the  switching  times  apparent  in 
Figures  5-8. 
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5.  The  actual  X-22  receives  Inputs  not  only  from  the  pilot 
but  from  at  least  two  sources  of  disturbances,  natural 
atmospheric  turbulence  and  synthetic  ship  wake  turbulence. 

In  the  flight  experiment  the  latter  disturbance  was  deli¬ 
berate  and  measured  while  the  former  is  random,  unmeasured , 
and  therefore  unknown. 

6.  The  pilot,  sensing  the  disturbances,  will  tend  to  generate 

a  control  which  minimizes  the  differences  between  the  actual 
X-22A  responses,  X^(t),  and  the  desired  X-22A  responses, 

Xp(t)  by  feeding  back  the  response  error  through  a  feedback 
gain  matrix,  Kp. 

6.4  Estimation  of  Pilot  Models.  Given  the  pilot  model  discussed  above 
and  described  in  Figures  10-12  it  is  possible  to  determine  the  unknown 
parameters  of  the  model  by  using  the  time  history  measurements  of 
actual  pilot/vehicle  performance  and  modern  state  and  parameter  esti¬ 
mation  techniques.  Efforts  by  the  authors  in  this  direction  are  con¬ 
tinuing. 

The  unknown  parameters  which  must  be  determined  from  the  flight 
data  are  listed  on  Figures  11  and  12.  It  should  be  noted  that  the 
switching  times  and  magnitudes  in  the  pilots  bang-bang  control  policy 
are  estimated  from  the  flight  data  and  are  not,  in  the  general  case, 
the  result  of  forcing  optimality.  Therefore,  the  pilot's  desired 
trajectory  model  may  not  be  optimal  in  any  sense  and  in  fact  may  not 
even  guarantee  exact  positioning  over  the  landing  pad.  If  this  is 
determined  to  be  a  problem  additional  pilot  integral  feedback  paths 
may  be  required  to  insure  proper  vehicle  positioning  with  respect 
to  the  actual  landing  pad. 
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The  extension  of  this  approach  to  the  multi -axis  task  can  be  accom¬ 
plished  by  identifying  "urgency  functions"  or  attention  switching  boundar¬ 
ies  from  the  flight  data.  These  "urgency  functions"  developed  by  Onstott 
and  Faulkner  in  Reference  3  allow  for  pilot  attention  switching  from  one 
axis  task,  eg  x-positioning,  to  another  axis  task,  eg.  y-positioning. 
Efforts  in  this  direction  are  planned. 

7.  Conclusions.  A  pilot  model  has  been  proposed  based  upon  a  preliminary 
analysis  of  hovering  X-22A  flight  data  performing  a  discrete  tracking  task 
in  the  presence  of  disturbances.  The  pilot  model  encompasses  both  a 
bang-bang  control  policy  for  amplitude  limited  gross  maneuvers  and  an 
error  feedback  model  for  disturbance  rejection.  Attempts  are  underway  to 
use  this  model  form  to  identify  the  unknown  parameters  of  the  model  from 
the  X--22A  flight  data  using  state  and  parameter  estimation  techniques. 
Extension  to  multi -axis  tasks  are  possible  by  identification  of  urgency 

or  attention  functions  from  the  flight  data. 
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TURBULENCE 


iTERAL  AXIS  FLIGHT  CONTROL  BLOCK  DIAGRAM 


Degrees 


FLIGHT  CONFIGURATION  208A 


DISTURBANCES 


PROPOSED  PILOT  MODEL 


Pilots  Internal  Desired  Trajectory  Model 


T&p(t)  =  -  6D(t)  +  6c(t-t0)  ,  6d(o)  =  0 

XD(t)  =  A  XD(t)  +  B  fiD(t)  ,  XD(o)  =  0 

Pilot  Output  Model 

xS(t)  =  -  6(t)  +  fic(t-t0)  -  Kp(XD(t-t0)  -  XA(t-t0)) 

+  m(t-t0)  ,  <5 (o) 

Vehicle  Model 

XA  =  A  XA(t)  +  Bfi(t)  +  CD(t)  ,  XA(o) 

Measurement  Equations  for  Parameter  and  State  Estimation 
yt)  =  XA(t)  +  nx(t)  +  b, 

«sm(t)  =  6(t)  +  nE(t)  +  b2 


Unknowns 


^C*  T >  ^p»  ^(o)»  XA(o) ,  b-j »  b2 


Figure  11 

Pilot  Model  Equations  of  Motion 
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ABSTRACT 


A  joint  NASA/Army  effort  to  develop  a  qualitative  assessment  of  simulator 
fidelity  is  underway  at  the  Ames  Research  Center.  The  approach  is  to  first 
obtain  a  highly  accurate  mathematical  model  and  then  couple  this  with  an  appro¬ 
priate  visual,  motion,  and  computation  system.  The  vehicle  being  used  for  this 
study  is  the  Army  Sikorsky  UH-60  Blackhawk  helicopter,  A  complex  math  model  has 
been  obtained  from  Sikorsky  and  programmed  for  real  time  operation  at  the  Ames 
Research  Center.  The  model  will  be  verified  by  comparing  with  flight  data  being 
obtained  in  a  special  flight  test  program  performed  by  the  Army  engineering 
flight  activity  at  Edwards  Air  Force  Base  in  the  period  September  1981  through 
June  1982.  As  a  corollary  to  the  testing  for  the  mathematical  model  parameter 
identification,  this  same  highly  instrumented  UH-60  will  also  be  used  to  obtain 
pilot  performance  data  to  use  in  the  simulation  fidelity  validation  process. 
Following  analysis  of  the  flight  data  these  simulations  will  be  performed  at  Ames 
and  an  assessment  made  of  the  range  of  validity. 
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ABSTRACT 

Procedures  for  identifying  pilot  dynamics  are  reviewed,  with 
emphasis  on  frequency-response  analysis  of  data  obtained  in 
well-controlled  test  environments.  This  paper  discusses  some 
theoretical  constraints  relevant  to  the  computation  of  operator 
describing  functions  and  remnant  spectra;  reviews  advantages  and 
limitations  of  frequency-domain  techniques  commonly  used;  and 
describes  the  use  of  sum-of-sines  inputs  to  enhance  the  measurement 
environment.  Procedures  for  identifying  pilot-related  parameters 
of  the  Optimal  Control  Model  are  reviewed. 


INTRODUCTION 

Techniques  for  measuring  pilot  response  behavior  in 
closed-loop  control  tasks  have  reached  an  advanced  state  of 
development  oyer  the  past  few  decades.  Proper  application  of  this 
technology,  combined  with  experimental  design  that  maximizes 
measurement  capabilities,  can  lead  to  reliable  and  accurate 
estimates  of  human  operator  behavior.  On  the  other  hand, 
ill-suited  techniques  or  unfavorable  measurement  situations  may 
yield  grossly  inaccurate  estimates  of  response  behavior. 

Measurement  of  pilot  response  is  usually  undertaken  to 
describe  response  behavior  in  a  particular  control  situation,  and, 
in  many  cases,  to  determine  a  general  set  of  rules  that  allows  one 
to  predict  pilot  response  in  a  variety  of  control  tasks.  The  end 
product  of  the  measurement/analysis  procedure  is  often  of  a  set  of 
coefficients  for  a  particular  pilot/vehicle  model. 
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Quantification  of  pilot  response  behavior  in  terms  of  model 
parameters  may  be  done  directly  or  through  an  intermediate  stage  of 
data  reduction.  The  one-stage  procedure  leads  directly  from 
experimental  time  histories  to  model  parameters;  the  two-stage 
procedure  first  transforms  the  information  contained  in  the  time 
histories  to  a  format  that  is  more  readily  interpretable  in  terms 
of  operator  response  strategy.  Model  analysis  is  then  performed  on 
the  transformed  data. 

The  bulk  of  this  paper  focuses  on  the  first  stage  of  the 
two-stage  analytical  procedure;  specifically,  the  transformation  of 
experimental  time  histories  to  frequency-response  measures  that 
relate  to  the  linear  portion  of  the  pilot's  response  strategy  (the 
describing  function)  as  well  as  to  the  stochastic  portion  of  the 
pilot's  control  response  (the  power  spectral  density  of  the  "pilot 
remnant") .  These  metrics  are  useful  for  testing  a  variety  of 
models,  including  both  "classical"  frequency-response  models  [1]  as 
well  as  the  optimal  control  model  for  pilot/vehicle  systems 
described  in  a  companion  paper  [2], 

Discussion  is  limited  to  situations  in  which  pilot  response 
behavior  is  representable  as  a  noisy,  time-invariant  linear 
process.  This  restriction  implies  that  (1)  the  controlled  plant  is 
representable  as  a  time-invariant  linear  system,  (2)  the  external 
forcing  function (s)  have  stationary  statistical  properties,  and  (3) 
the  pilot  has  been  sufficiently  motivated  and  trained  to  the  stage 
where  he  has  stabilized  upon  a  response  strategy  appropriate  to  the 
control  task. 

The  remainder  of  this  paper  is  organized  into  three  major 
sections.  The  first  section  reviews  certain  theoretical  aspects  of 
frequency  response  analysis  and  points  out  the  advantages  of  having 
external  inputs  available  for  measurement.  Computational 
techniques  appropriate  to  inputs  continuous  in  frequency  as  well  as 
discrete-frequency  inputs  (i.e.,  sum-of-sinusoids)  are  treated  in 
the  subsequent  section.  The  final  major  section  provides  a  brief 
description  of  a  technique  for  identifying  and  statistically 
testing  pilot-related  parameters  of  the  optimal  control  model. 

THEORETICAL  ANALYSIS 

In  this  section  we  consider  some  of  the  limitations  and 
sources  of  error  associated  with  measurement  of  pilot  response 
characteristics  in  closed-loop  control  tasks.  Without  presupposing 
any  specific  measurement  techniques  at  this  stage,  we  shall  assume 
that  cross  correlations  and/or  cross  power  spectral  densities  (PSD) 
are  obtainable  between  any  two  measured  quantities.  We  show  how 
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analysis  is  influenced  by  system  complexity  (single-  versus 
multi-variable  systems) ,  by  the  availability  of  the  external  input 
for  measurement,  and  by  the  presence  or  absence  of  certain 
constraints  on  the  computation  of  cross  PSD. 

Analysis  of  Single-Variable  Systems 

Let  us  consider  the  task  of  analyzing  the  single-variable 
feedback  control  system  diagrammed  in  Figure  1.  The  objective  is 


i(t) 


r(t) 


i  ( t )  =  INPUT  FORCING  FUNCTION 
e  ( t )  =  SYSTEM  ERROR 
c  ( t )  =  OPERATOR'S  CONTROL  INPUT 
r  ( t )  =  OPERATOR  REMNANT 
H  ( jtt>)  =  OPERATOR  DESCRIBING  FUNCTION 
V(jw)  =  CONTROLLED-ELEMENT  TRANSFER 
FUNCTION 


Figure  1.  Diagram  of  Single  Variable  System 


to  estimate  ..the  pilot's  transfer  function  H  as  well  as  the  power 
spectral  density  of  the  "remnant"  r(t).  The  pilot's  remnant  is 
assumed  to  be  linearly  independent  of  the  external  forcing 
function,  and  is  not  a  directly  measurable  quantity.  No 
assumptions  are  made  with  regard  to  the  spectral  content  of  the 
remnant . 
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The  pilot's  transfer  characteristic  is  not,  strictly  speaking, 
a  fixed  quantity,  because  the  pilot  adjusts  his  strategy  according 
to  the  nature  of  the  control  task.  Accordingly,  manual  control 
research  has  adopted  the  term  "describing  function"  to  indicate  the 
pilot's  linear  response  strategy  in  a  specific  control  situation. 
We  shall  adopt  this  terminology  in  the  remainder  of  the  paper. 

The  theoretical  development  presented  below  makes  extensive 
use  of  the  following  properties  of  linear  systems: 

$xy(jt0)  =  H(joj)  *  $Xx(u)) 

$*xy(j^)  =  H*(jw)  •  $xx(w)  (1) 

4yy(j“)  =  |  H  ( joi)  I  2  $xx(oO 

where  ®Xy(ja))  is  the  Fourier  transform  of  the  cross-correlation 
function  relating  input  x(t)  variable  to  output  variable  y(t)  , 
H(jco)  is  the  linear  system  function,  and  the  operation  (*) 
indicates  complex  conjugate.  For  economy  of  notation,  the 
frequency  argument  (jto)  is  omitted  from  the  following  analysis. 

The  following  cross  spectral  density  functions  are  derived 
from  the  above  relationships: 
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Because  of  the  assumed  linear  independence  between  the 
external  forcing  function  and  operator  remnant,  the  cross  PSD  term 
should  be  zero.  Even  when  these  signals  are  truly  independent, 
however,  limitations  on  the  measurement  interval  may  introduce  an 
apparent  correlation.  As  we  show  later,  this  apparent  correlation 
can  be  minimized  through  appropriate  experimental  and  analytical 
techniques . 


The  task  of  identifying  pilot  response  characteristics  is 
greatly  facilitated  when  the  external  inputs  are  available  for 
measurement,  as  is  usually  the  case  in  ground-based  simulation 
experiments.  These  signals  are  often  absent  for  data  obtained 
in-flight,  however.  In  the  following  discussion  we  first  summarize 
some  of  the  problems  of  analyzing  data  bases  in  which .  external 
inputs  are  absent,  and  we  then  show  how  measurement  accuracy  can  be 
improved  when  these  inputs  are  available. 


External  Inputs  Unavailable.  Unconstrained  Correlations 

Let  us  consider  the  measurement  situation,  diagrammed  in 
Figure  2a,  in  which  only  the  system  error  and  pilot's  control  input 
are  available  for  measurement.  The  external  input  (e.g.,  simulated 
wind  gust,  target  motion,  etc.)  is  assumed  unavailable.  The 
"remnant"  input  r ( t )  is  internal  to  the  pilot  and  is  therefore  not 
directly  measurable.  We  first  consider  measurement  capabilities 
when  no  constraints  are  placed  on  the  measured  cross  correlations. 

Application  of  the  properties  of  equation  (1)  to  the  system 
shown  in  Figure  2a  yields  the  following  relationships: 
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b)  MODEL  ANALYSIS 
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Figure  2.  Measurement  Situation  with  External  Input 
Unavailable 

In  the  ideal  measurement  situation,  there  is  no  cross 
correlation  between  tracking  error  and  pilot  remnant,  in  which  case 
®fe  =  0*  Since  r(t),  and  thus  Sre,  are  not  measurable,  one  has 
little  choice  but  to  assume  ideal  measurement  conditions  and  to 
utilize  the  following  estimation  strategy  based  on  the 
relationships  shown  in  equations  (9  and  10). 
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where  the  symbol  (A)  signifies  an  estimated  quantity.  In  general, 
however,  ffire  is  not  zero,  and  the  estimated  pilot  response 
variables  are  related  to  the  measurable  quantities  as  follows: 
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Using  the  relationships  of  equations  (6-8) ,  we  obtain  the 
following  expression  for  the  estimated  describing  function  in  terms 
of  external  signals  driving  the  system: 
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Thus,  where  pilot  remnant  is  low,  a  good  estimate  to  the 
pilot's  describing  function  is  obtained.  Where  remnant  is 
relatively  large,  however,  the  estimate  tends  toward  the  negative 
inverse  of  the  vehicle  transfer  function  and  is  unrelated  to  pilot 
response  strategy. 

A  similar  situation  arises  with  respect  to  the  estimate  of 
pilot  remnant,  which  is  related  to  external  signals  by 
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Again,  a  reliable  estimate  is  obtained  when  remnant  is  low;  but 
for  large  remnant,  the  estimate  of  the  pilot  remnant  spectrum 
approaches  the  spectrum  of  the  external  input  and  bears  no 
relationship  to  pilot  response  behavior. 

To  summarize  the  theoretical  analysis  so  far,  when  pilot 
response  parameters  are  based  only  on  measurements  of  system  error 
and  pilot  control  input,  errors  in  estimating  both  the  pilot 
describing  function  and  remnant  spectrum  are  likely  if  the  total 
correlation  between  error  and  control  is  considered.  Such  errors 
arise  largely  from  the  failure  to  consider  cause  and  effect. 
Specifically,  even  though  pilot  remnant  is  theoretically 
independent  of  the  other  external  inputs  to  the  system,  a  "backward 
correlation"  between  system  error  and  remnant  will  generally  be 
measured  due  to  the  response  of  the  system  to  the  pilot's  input. 
This  correlation  distorts  the  estimates  of  pilot  response 
parameters,  sometimes  to  the  point  that  the  estimated  describing 
function  does  not  reflect  a  physically  realizable  system.  As 
discussed  below,  improved  estimates  of  pilot  describing  function 
and  remnant  are  possible  in  high-remnant  situations  through 
measurement  techniques  that  reduce  the  influence  of  the  "backward 
correlation" . 

External  Inputs  Unavailable.  Constrained  Correlation 

Model-matching  techniques  are  often  employed  to  impose  desired 
cause  and  effect  relationships  on  the  analysis  procedure  when 
external  inputs  are  not  available  for  measurement.  As  shown  in 
Figure  J2b,  one  obtains  the  difference  between  the  measured  pilot 
input  c(t)'-  and  the  signal  c(t)  predicted  by  filtering  the  error 
signal  through  some  model,  defined  by  transfer  function  M(jw).  The 
model  M  is  adjusted  to  minimize  the  matching  error  e2(t),  where 
C  ( t )  is  the  difference  between  the  actual  and  predicted  control 
signals.  The  model  M  is  then  taken  as  the  estimate  of  the  pilot 
describing  function. 

The  following  spectral  quantities  are  derived  for  C: 
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even  in  the  presence  of  sizeable  cross  correlations  between  error 
and  remnant.  If  the  matching  error  is  not  negligible,  however,  it 
will  introduce  errors  into  the  estimated  remnant  spectrum  as 
indicated  in  Equation  (16)  . 

A  combination  of  non-negligible  matching  error  and 
remnant-related  tracking  error  may  also  introduce  a  bias  in  the 
estimate  _of  the  operator's  describing  function.  Because  the 
quantity  M*ffire  +  M$*re  (a  real  quantity)  is  not  constrained  to  be 
non-negative,  it  may  be  possible  to  obtain  minimum  C2(t) 
(equivalently,  minimum  integral  of  the  spectral  density  ®cc)  for 
M^H . 


The  effects  of  the  cross  terms  in  ffigg  can  be  minimized  if  the 
model  M  is  constrained  to  be  physically  realizable  (i.e.,  does  not 
respond  to  inputs  that  have  not  yet  occurred) .  Estimation 
reliability  is  further  enhanced  if  the  effective  correlation  time 
of  the  remnant  signal  (i.e.,  the  reciprocal  of  the  bandwidth)  is 
less  than  the  delay  associated  with  the  pilot's  response  [3,4]. 

By  imposing  the  above  constraints  on  his  measurement 
procedure,  Wingrove  [4]  has  been  able  to  obtain  good  results  from 
in-flight  and  laboratory  data  in  the  absence  of  a  measurable 
forcing  function.  The  reader  is  referred  to  Wingrove  for  further 
discussion  of  these  techniques. 

Although  minimizing  the  effects  of  the  cross  terms  in  equation 
(16)  reduces  one  source  of  estimation  error,  this  alone  does  not 
guarantee  an  accurate  estimate  of  operator  describing  function  and 
remnant  spectrum.  Implementation  of  a  model-matching  procedure 
imposes  a  particular  structure  on  operator  response  behavior;  to 
the  extent  that  this  structure  is  inadequate,  estimation  errors 
will  be  introduced. 
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The  measurement  situation  is  greatly  enhanced  when  the 
external  inputs  are  available  for  measurement:  reliable  estimates 
of  both  the  human  operator  describing  function  and  remnant  spectrum 
may  be  obtained  without  cause-and-ef f ect  constraints  placed  on  the 
cross  correlations.  Dividing  Eg.  (8)  by  Eg.  (6),  we  obtain 
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Since  a>ir  is  theoretically  zero  (and,  in  practice,  approaches  zero 
for  a  sufficiently  long  measurement  interval) ,  the  ratio  of  cross 
spectra  shown  here  affords  a  good  approximation  to  the  pilot 
describing  function.  Therefore, 
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A  good  approximation  to  the  closed-loop  remnant  spectrum  is 
given  by 

®cc  =  ®cc  “  I  ®icl  2/S>ii  (20) 

From  eguations  2-7,  we  show  that,  in  terms  of  input-  and 
remnant-correlated  guantities,  this  expression  is  eguivalent  to 
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This  closed-loop  measure  can  be  converted  to  an  estimate  of  the 
open-loop  (injected)  remnant  by  scaling  with  |1+HV|2. 

As  can  be  seen  from  the  above  development,  use  of  the  external 
forcing  function  in  the  analysis  procedure  avoids  measurement 
errors  introduced  by  large  pilot  remnant.  Accurate  estimates  of 
pilot  describing  function  and  remnant  may  be  obtained  in  the 
presence  of  large  <Err,  provided  the  apparent  input-remnant 
correlation  is  sufficiently  small.  A  technique  for  minimizing 
ffiir  is  reviewed  later  in  this  paper. 

Analysis  of  Multivariable  Systems 

Since  considerations  of  length  preclude  a  detailed  treatment 
of  multivariable  systems,  we  present  here  a  very  brief  summary  of 
the  analysis  documented  by  Levison  [3],  The  (matrix)  pilot 
describing  function  is  estimated  from  the  cross  PSD's  as  follows: 

£  =  -fiic  •  Sie"1  (22) 

where  H  contains  elements  relating  each  "error"  quantity  to  each 
control  input,  and  the  cross  PSD's  &ic  and  relate  each  input  to 
each  control  and  each  error  variable,  respectively. 

In  order  for  this  computation  to  be  valid,  lie  must  be 
invertible.  This  imposes  the  following  rather  severe  restrictions: 
(1)  there  must  be  as  many  external  forcing  functions  as  there  are 
error  (i.e.,  display)  variables;  (2)  these  forcing  functions  must 
be,  or  be  transformations  of,  linearly  independent  processes,  and 
(3)  the  output  variables  must  be  differentially  influenced  by  the 
forcing  functions. 

Even  when  these  conditions  are  met,  one  cannot  in  general 
compute  the  transfer  element  Hj^  from  only  the  PSD's  involving  the 
jth  control  variable  and  theJ  kth  display  variable;  the  matrix 
operation  shown  above  must  be  carried  out  to  yield  readily 
interpretable  measures  of  pilot  performance.  Sample  multivariable 
control  systems  are  analyzed  in  Reference  3  to  illustrate  this 
point.  Also  derived  in  that  reference  is  a  (rather  complex) 
expression  for  the  vector  remnant  process  fi.rr. 

COMPUTATIONAL  TECHNIQUES 

Issues  related  to  frequency-response  analysis  techniques  are 
reviewed  in  this  section.  We  briefly  review  certain  important 
aspects  of-  Fourier-series  analysis,  and  we  then  consider 
continuous-frequency  and  sum-of-sinusoids  measurement  techniques. 
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Fourier  Analysis 

The  advent  of  computationally-ef f icient  fast-Fourier  transform 
techniques  [5]  has  greatly  facilitated  analysis  of  data  obtained  in 
manual  control  experiments,  and  the  existence  of  these  techniques 
has  encouraged  the  design  of  experiments  in  which  steady-state 
frequency-domain  analysis  is  applicable. 

Application  of  Fourier  series  methods  assumes  that  a  time 
series  of  finite  length  may  be  treated  as  if  it  were  periodic.  (We 
do  not  necessarily  assume  that  the  data  are  actually  periodic,  but 
rather  that  we  can  obtain  the  desired  information  using  analysis 
procedures  designed  for  periodic  signals.) 

If  a  signal  is  periodic  in  time  T,  we  define  a  fundamental  or 
"base"  frequency  as  ui0  =  2  7t/T,  and  we  note  that  all  frequency 
components  of  such  a  signal  must  be  harmonics  of  this  base 
frequency.  The  cross  PSD  is  given  as 


$  =  Y (k) X* (k) 

xy 


(23) 


where  k  is  an  integer,  X(k)  and  Y(k)  are  Fourier  coefficients 
defined  at  frequency  kw0,  and  the  overstrike  indicates  statistical 
expectation.  Fourier  coefficients  and  spectral  quantities  are 
defined  for  both  positive  and  negative  frequencies  (i.e.,  positive 
and  negative  values  of  "k")  . 

The  quantity  $xy  is  most  properly  termed  a  "cor relogram" , 
rather  than  a  "spectrum"  when  analyzing  a  finite-time  sample  of  a 
random  process.  Although  this  quantity  is  often  computed  to  obtain 
an  estimate  of  the  spectrum  of  a  random  process,  the  standard 
deviation  of  the  error  associated  with  such  an  estimate  is  large  — 
on  the  order  of  the  expected  value  of  the  estimate  [5].  The 
effects  of  this  error,  however,  can  be  considerably  reduced  through 
appropriate  averaging  techniques.  Additional  improvement  in 
estimation  accuracy  can  be  obtained  if  inputs  are  constructed  as 
sums  of  sinusoids,  in  which  case  the  input  correlated  portion  of 
the  signal  is  truly  periodic. 

In  the  remainder  of  this  section  we  consider  the  analysis  of  a 
single-variable  control  task  as  diagrammed  in  Figure  1.  We  first 
consider  inputs  that  are  continuous  in  frequency,  then  we  show  the 
computational  advantages  of  using  sum-of-sines  inputs. 
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Inputs  Continuous  in  Frequency 

Assume  that  the  tracking  input  i(t)  is  continuous  in  frequency 
in  the  regions  where  frequency-response  measures  are  desired.  From 
the  relationships  of  Eqs.  (19)  and  (20),  plus  the  property  of 
Eq.(23),  we  derive  the  following  describing  function  and  remnant 
estimates  in  terms  of  measurable  quantities: 


H 


$ . 
ie 


Cl* 

El* 


A 


1C 


2 


(24) 


where  I,  E,  and  C  are  the  Fourier  coefficients  of  the  input,  error, 
and  control  signals,  respectively,  at  a  specific  frequency  ka)0' 
fficc  denotes  the  closed-loop  remnant  spectrum;  i.e.,  the  portion  of 
ther control  power  related  to  the  remnant  input.  (For  notational 
convenience,  we  shall  omit  the  frequency  argument  for  Fourier 
coefficients . ) 

These  estimates  are  related  to  system  variables  as  follows: 
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Thus,  as  the  number  of  measurements  available  for  averaging 
increases,  the  quantity  RT*  tends  toward  zero  (because  of  the 
presumed  lack  of  linear  correlation  between  the  external  input  and 
operator  remnant  process)  ,  and  the  estimates  approach  their  true 
values.  The  quantities  H  and  V  are  assumed  constant  and  are 
therefore  extracted  form  the  averaging  process.  If  these 
quantities  are  not  constant,  then  an  average  H  would  be  estimated. 

Averaging  must  be  performed  on  the  individual  cross  spectral 
quantities,  rather  than  on  estimates  of  H  and  3>cc  obtained  from 
individual  sets  of  measurements,  because  one  cannot  distinguish 
between  remnant-related  and  input-correlated  signal  power  from 
single  measurements  of  I,  E,  and  C.  Averaging  may  be  done  either 
across  frequency  (i.e.,  average  measurements  obtained  from 
neighboring  frequencies)  ,  across  an  ensemble  (average  measurements 
at  a  single  frequency  obtained  from  repeated  trials  of  the 
experiment) ,  or  a  combination  of  both.  In  any  case,  we  must  make 
the  assumption  that  the  system  function  H  is  constant  across  the 
frequency  band  and  from  trial  to  trial.  This  constraint  requires 
that  the  frequency  band  across  which  averages  are  taken  be 
relatively  small.  (In  practice,  this  author  has  used  bandwidth  of 
about  1/4  octave  for  performing  within-trial  averaging  of  spectral 
quantities . ) 

Because  the  quantities  to  be  averaged  are  cross-spectral 
measures  and  not  the  time  signals  or  Fourier  coefficients,  one  may 
use  different  inputs  from  trial  to  trial.  Statistics  of  the  input 
signals  should  be  kept  constant,  however,  to  minimize  trial  to 
trial  deviation  in  the  pilot's  response  strategy. 

The  number  of  data  points  required  to  obtain  a  reliable 
estimate  of  H  decreases  with  the  relative  amount  of  remnant 
present.  Equation  (25)  shows  that  we  get  a  "perfect"  estimate  of  H 
if  zero  remnant  is  present.  If  substantial  remnant  is  present,  the 
measurement  situation  can  be  aided  by  concentrating  input  power  in 
selected  frequency  bands  and  by  averaging  across  repeated  trials  by 
the  same  subject  and  across  subjects. 

The  task  of  obtaining  reliable  estimates  of  remnant  power 
appears  to  be  substantially  more  difficult  than  that  of  estimating 
the  describing  function.  If  the  remnant  power  density  is  very 
small  (which  it  typically  is  for  control  response  at  low 
frequencies  under  ideal  tracking  conditions) ,  a  numerical  problem 
may  arise.  As  Equation  (24)  shows,  remnant  power  is  computed  as 
the  difference  between  total  and  correlated  power.  If  these 
estimates  are  nearly  equal,  precision  may  be  lost  due  to  roundoff 
errors . 
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Errors  in  estimating  remnant  also  arise  from  the  measured 
(apparent)  correlation  between  the  remnant  and  the  input  variable. 
Ideally,  sufficient  measurements  would  be  averaged  so  that  the 
effect  of  the  quantity  RT*  becomes  negligible.  This  is  usually  not 
a  severe  problem  for  high-frequency  measurements,  where  the 
averaging  window  (say,  1/4  octave)  typically  includes  a  substantial 
number  of  frequency  intervals.*  A  more  serious  problem  exists  in 
obtaining  accurate  estimates  of  remnant  at  low  frequencies,  where 
the  number  of  frequency  intervals  per  window  is  smaller. 

As  described  below,  measurement  inaccuracies  arising  from  the 
cross-spectral  quantity  KT*  can  be  substantially  reduced  by  the  use 
of  an  external  forcing  function  constructed  as  a  sum  of  sinusoids. 

Sum-of-Sines  Inputs 

An  appropriate  combination  of  sinusoids  can  provide  a  tracking 
input  that  appears  to  be  stochastic  to  the  subject  but  which 
overcomes  many  of  the  measurement  difficulties  associated  with 
inputs  that  are  continuous  in  frequency.  This  type  of  input  allows 
us  to  model  (and  otherwise  interpret)  pilot  behavior  as  a  response 
to  an  unpredictable  input,  while  providing  the  computational 
benefits  of  the  highly  structured  sum  of  sinusoids. 

If  each  component  frequency  is  harmonically  related  to  the 
reciprocal  of  the  run  length,  and  if  each  component  sinusoid  is 
generated  without  distortion,  then  Fourier  analysis  of  the  input 
signal  will  contain  power  only  at  the  nominal  input  frequencies. 
Since  there  is  no  input  power  at  non-input  frequencies,  any 
measurement  of  power  at  non-input  frequencies  (provided  it  is  above 
irreducible  system  noise)  is  by  definition  related  to  operator 
"remnant".  In  order  to  estimate  the  remnant  power  density  in  the 
vicinity  of  a  nominal  input  frequency,  we  simply  average  the  power 
measurements  over  a  frequency  band  on  either  side  of  (but  not 
including)  the  input  frequency.  Estimates  of  $cc  can  be  averaged 
both  across  subjects  and  across  trials  for  the  same  subject  in 
order  to  improve  accuracy. 

With  a  sum-of-sines  input,  no  averaging  procedure  is  required 
to  obtain  estimates  of  the  controller's  describing  function,  and 
the  expression  of  Equation  (24)  reduces  to 


*  The  number  of  "frequency  intervals"  contained  in  a  frequency 
rangeAw  is  equal  to  Aoj/oj0.  Thus,  estimation  accuracy  can  be 
enhanced  by  increasing  the  run  length,  which  decreases  the  base 
frequency  and  provides  more  frequency-response  measurements  for  a 
given  frequency  range. 
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H  =  Cl */EI *  =  C/E 


(26) 


That  is,  the  describing  function  is  simply  the  ratio  of  the  Fourier 
coefficients  of  the  control  and  error'  signal  at  each  input 
frequency. 

Now,  since  the  spectral  measurement  obtained  at  a  given  input 
frequency  contains  both  remnant  power  as  well  as  power  truly 
correlated  (i.e.,  linearly  related)  to  the  tracking  input, 
computations  of  the  Fourier  coefficients  C  and/or  E  may  be 
contaminated  by  operator  remnant,  introducing  errors  in  the 
estimation  of  the  operator  1 s  describing  function.  Typically,  these 
errors  are  small,  at  least  for  laboratory  tracking  tasks.  By  using 
a  sum-of-sines  input,  we  concentrate  the  input-correlated  power  at 
a  few  selected  frequencies.  Since  remnant  is  continuous  in 
frequency  (an  assumption  confirmed  in  numerous  laboratory  tracking 
studies) ,  input-correlated  power  at  a  given  input  frequency  is 
usually  considerably  greater  than  remnant  power  overlapping  the 
same  frequency. 

An  indication  of  the  reliability  of  the  describing  function 
estimate  at  a  given  frequency  can  be  obtained  from  a  comparison  of 
the  power  measured  at  that  frequency  to  the  neighboring  remnant 
power.  If  the  power  at  the  input  frequency  is  sufficiently  greater 
than  the  surrounding  remnant  power  (say,  by  7  dB  or  more)  for  both 
error  and  control  signals,  estimates  of  C,E,  and  H  may  be 
considered  reliable.  Otherwise,  the  estimate  of  H  should  be 
considered  unreliable  and  should  not  be  used  as  a  basis  for 
modeling  or  otherwise  characterizing  human  operator  behavior. 

In  summary,  use  of  a  sum-of-sines  input  provides' increased 
accuracy  in  the  estimation  of  operator  describing  functions  and 
operator  remnant  spectra.  Describing  functions  can  be  obtained 
without  averaging  cross  spectral  quantities,  and  estimates  of  the 
remnant  spectruri  are  free  from  errors  introduced  by  the  (apparent) 
cross  correlation  between  input  and  remnant  that  is  present  when 
inputs  are  continuous  in  frequency.  Comparison  of  spectral 
measures  at  input  and  non-input  frequencies  provides  an  indication 
of  the  reliability  of  the  describing  function  estimate. 

Use  of  sum-of-sines  inputs  necessarily  restricts  estimation  of 
operator  describing  function  to  selected  frequencies.  In  most 
control  situations,  however,  one  can  obtain  measurements  at  a 
sufficient  number  of  frequencies  to  adequately  characterize 
operator  response  behavior. 


692 


In  order  to  provide  maximum  analytical  power,  sum-of-sines 

inputs  designed  for  manual  control  studies  should  meet  the 

following  criteria: 

a.  The  measurement  interval  should  contain  an  integral  number  of 
cycles  of  each  component  sinusoid.  (Equivalently,  each 
component  frequency  should  be  harmonically  related  to  the 
reciprocal  of  the  run  length.) 

b.  A  sufficient  number  of  sinusoids  should  be  combined  with  random 
phasing  so  that  the  input  is  random-appearing  tb  the  subject. 

c.  The  effective  bandwidth  of  the  input  should  be  as  wide  as 
possible  to  maximize  the  range  of  valid  measurements,  but  not 
so  wide  as  to  unrealistically  affect  tracking  behavior. 

d.  Amplitudes  of  component  sinusoids  should  be  selected  so  that 
the  input  approximates  some  desired  theoretical  spectral 
density  function. 

e.  Phase  relationships  should  be  readjusted  from  trial-to-trial 
(for  a  given  subject)  to  prevent  learning  of  the  input. 


The  reader  is  referred  to  Levison  [3]  for  elaboration  of  these 
criteria  as  well  as  for  procedures  for  generating  sum-of-sines 
inputs;  for  converting  discrete  spectral  estimates  to  equivalent 
continuous  power  spectral  densities,  sampling  and  filtering  analog 
signals;  and  for  correcting  for  measurement  delays. 

Although  the  foregoing  discussion  has  been  limited  to 
single-variable  tracking  tasks,  sum-of-sines  inputs  may  also  be 
used  to  estimate  the  matrix  transfer  function  when  the  operator  is 
provided  with  more  than  one  display  variable.  As  noted  earlier, 
there  must  be  as  many  independent  tracking  inputs  as  there  are 
displayed  variables.  Analysis  and  design  procedures  for  tasks 
utilizing  multiple  sum-of-sines  inputs  are  discussed  in  Reference 
[6]  . 

IDENTIFICATION  OF  PILOT-RELATED  MODEL  PARAMETERS 

The  methods  described  above  for  reducing  time  histories  to 
frequency-response  measures  may  be  considered  the  first  step  of  a 
two-stage  procedure  for  identifying  pilot  response  parameters.  In 
this  section  we  review  the  second  step:  the  derivation  of  pilot 
model  parameters  from  the  reduced  data. 
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Discussion  is  limited  to  the  "optimal  control  model"  for  the 
human  operator  in  steady-state  control  tasks.  Typical  independent 
—  or  "pilot-related"  —  parameters  to  be  identified  are  time 
delay,  motor  time  constant,  (equivalently,  a  "cost"  penalty  on 
rate-of-change  of  control) ,  motor  noise  covariance,  and  an 
observation  noise  covariance  for  each  perceptual  input  variable 
used  by  the  operator.  Readers  unfamiliar  with  this  model  are 
directed  to  a  companion  paper  [2] .  Additional  details  on  parameter 
identification  are  given  in  [7,8]  . 

We  first  review  the  quasi-Newton  (QN)  minimization  procedure 
recently  developed  for  identifying  model  parameters,  and  then 
summarize  a  qualitative  scheme  for  evaluating  the  significance  of 
task-related  changes  in  parameter  values. 

Review  of  the  Quasi-Newton  Identification  Procedure 

The  Basic  Minimizat i on  Procedu re 

Consider  the  task  of  adjusting  model  parameters  to  minimize  a 
scalar  matching  error  where  each  element  ei  of  the  column 
vector  e.  is  the  difference  between  the  ith  measured  data  point  and 
the  corresponding  model  prediction,  and  each  element  w^  of  the 
diagonal  matrix  H.  is  a  weighting  coefficient.  In  a  given 
application,  the  matching  error  J  will  correspond  to  a  particular 
choice  of  parameter  values  p.  The  objective  of  the  search 
procedure  is  to  find  a  new  parameter  set  p+Ap  such  that  J  is 
minimized . 

To  implement  the  search  scheme,  we  initially  assume  that  model 
predictions  (and,  therefore,  prediction  errors)  vary  linearly  with 
model  parameters.  Thus,  Ajg.  =  Q'Ap,  where 

q  (  i  ,  j )  =3ej/9pi  (27) 
Solving  for  minimum  J  as  a  function  of  Ap,  we  obtain 

Ap  =  -[£»'  ]_1Qii£.  (28) 

Now,  since  model  input/output  relationships  are  seldom  totally 
linear,  two  or  more  iterations  of  the  procedure  are  required  until 
some  convergence  criteria  are  satisfied.  In  some  cases,  the 
parameter  change  computed  as  shown  in  Eq(28)  will  yield  a  scalar 
matching  error  greater  than  the  starting  value.  Therefore,  it  is 
often  useful  to  augment  the  minimization  procedure  described  above 
with  a  line-search  scheme  to  optimize  the  magnitude  of  Ap. 
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Implementation  for  Manual  Control  Studies 

Application  of  the  QN  method  for  analysis  of  human  operator 
performance  in  continuous  control  tasks  has  been  reported  by 
Lancraft  and  Kleinman  [9].  Described  below  is  a  revised 
implementation  that  has  been  used  by  this  author  in  recent  manual 
control  studies. 


Two  criteria  must  be  defined  in  order  to  apply  the 
identification  procedure:  (1)  a  definition  of  a  scalar  matching 
error  to  be  minimized  by  the  QN  scheme,  and  (2)  convergence 
criteria  to  determine  when  the  minimum  modeling  error  has  been 
approached  sufficiently  closely  to  justify  termination  of  the 
minimization  procedure. 


Matching  error  is  similar  to  that  used  by  Lancraft  and 
Kleinman: 
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where  N  is  the  number  of  valid  measurements  in  the  jth  measurement 
group;  G,P,  are  the  gain  (dB)  and  phase  shift  (degrees)  of  the  ith 
describing  function  point  to  be  matched;  R  is  the  corresponding 
control-stick  "remnant"  measurement  (dB) ;  and  S  is  the  ith  variance 
score  to  be  matched  (units  different  for  different  tracking 
variables)  .  0  indicates  standard  deviation  of  an  experimental 
data  point,  and  the  symbol  (*)  indicates  a  model  prediction. 

Inclusion  of  the  experimental  deviations  in  the  scalar 
modeling  error  allows  each  error  component  to  be  weighted  inversely 
by  the  reliability  of  the  data.  To  prevent  the  matching  criterion 
from  giving  excessive  weights  to  variables  that  have  very  low 
experimental  variability,  the  following  minimum  standard  deviations 
are  imposed:  0.5  dB  for  magnitude  and  remnant,  3  degrees  for 
phase,  and  5%  for  the  ensemble  mean  for  variance  scores. 
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Weighting  inversely  by  standard  deviation  also  converts  each 
error  term  into  a  dimensionless  number,  thereby  allowing 
accumulation  of  matching  errors  into  a  single  metric.  Thus,  the 
matching  error  defined  in  Eq(29)  approximates  the  average  number  of 
standard  deviations  of  mismatch.  A  numerical  score  of  J=4  reflects 
an  average  modeling  error  of  1  standard  deviation  (i.e.,  an  average 
error  of  unity  per  measurement  group) . 

The  minimization  procedure  is  terminated  when  the  following 
conditions  jointly  obtain  for  two  successive  iterations:  (1) 
reduction  of  the  matching  error  by  less  than  0.5%,  and  (2)  changes 
in  all  identified  parameters  by  less  than  2%. 

A  number  of  modifications  have  been  made  to  the  original 
implementation  in  order  to  improve  computational  efficiency. 
First,  the  search  is  performed  on  the  logarithms  of  the  parameters. 
This  transformation  modestly  increases  the  degree  of  linearity 
between  model  parameters  and  model  outputs,  and  it  prevents  the 
assignment  of  out-of-bounds  (i.e.,  negative)  values  to  parameters 
during  the  course  of  the  search.  Second,  in  order  to  minimize 
numerical  difficulties  with  inversion  of  the  expression  0W0 ' ,  we 
omit  from  the  search  procedure  (i.e.,  keep  fixed),  at  a  given 
iteration,  any  parameter  having  a  negligible  influence  on  the 
matching  error.  In  addition,  to  reduce  the  chance  of  convergence 
to  a  local  minimum  appreciably  removed  from  the  global  minimum,  an 
individual  parameter  is  allowed  to  undergo  no  more  than  a  ten-fold 
increase  or  decrease  from  one  iteration  to  the  next. 

Finally,  a  binary  section  scheme  is  employed  to  prevent 
divergence  of  the  QN  scheme  due  to  nonlinear  relationships  between 
model  inputs  and  outputs.  If  necessary,  binary  section  is  repeated 
until  (1)  matching  error  is  reduced  from  one  iteration  to  the  next, 
or  (2)  until  four  attempts  fail  to  reduce  matching  error,  at  which 
point  the  minimization  scheme  is  terminated.  Further  details 
regarding  implementation  have  been  documented  by  Levison  [7] . 

As  is  true  with  any  numerical  search  procedure,  the 
probability  of  convergence  to  a  global  minimum  is  enhanced  by  the 
selection  of  an  initial  set  of  model  parameters  that  are  close  to 
the  optimal  set.  The  following  rules  for  initializing  model 
parameters  appears  to  yield  good  results  with  the  QN  procedure: 
(1)  cost  of  control  rate  such  that  motor  time  constant  =  0.1 
seconds;  (2)  time  delay  =  0.2  seconds;  (3)  observation  noise 
covariance  to  achieve  a  noise/signal  ratio  of  -20  dB  for  each 
perceptual  variable  assumed  to  be  utilized  by  the  operator;  and  (4) 
motor  noise  covariance  to  achieve  a  noise  signal  ratio  of  -50  dB, 
normalized  with  respect  to  control-rate  variance. 
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Significance  Testing 


If  independent  model  parameters  are  observed  to  vary  across 
tasks,  or  to  vary  for  a  given  pilot  as  a  function  of  training,  one 
may  wish  to  test  for  the  statistical  significance  of  the  observed 
parameter  variations.  One  approach  is  to  test  the  hypothesis  that 
the  various  data  sets  can  be  modeled  by  the  same  set  of  model 
parameters.  Failure  to  support  this  null  hypothesis  indicates  that 
parameter  differences  are  also  significant. 

A  cross-comparison  scheme  has  been  developed  and  tested 
against  data  obtained  in  manual  control  studies.  This  scheme, 
which  employs  the  empirical  sensitivity  test  described  below, 
provides  a  qualitative  significance  test  on  parameter  differences 
obtained  from  pairs  of  experimental  conditions. 

Assume  that  model  parameters  have  been  identified  from  two 
data  sets  corresponding  to,  say,  the  "baseline"  and  "test" 
experimental  conditions;  our  task  is  now  to  test  the  null 
hypothesis  that  a  single  set  of  model  parameters  provides  a 
near-optimal  match  to  the  baseline  and  test  data.  To  perform  this 
test,  we  first  identify  the  following  three  sets  of  pilot 
parameters:  (1)  the  set  that  best  matches  the  baseline  data,  (2) 

the  set  that  best  matches  the  test  data,  and  (3)  the  set  that 
provides  the  best  joint  match  to  the  baseline  and  test  data.  For 
convenience,  we  shall  refer  to  the  parameters  identified  in  step  3 
as  the  "average  parameter  set". 

We  next  compute  the  following  four  matching  errors: 

J (B,B)  =  matching  error  obtained  from  baseline  data,  using 

parameters  identified  from  baseline  data  (i.e.,  best 
match  to  baseline  data) . 

J(B,A)  =  matching  error  obtained  from  baseline  data,  using  the 
average  parameter  set. 

J(T,T)  =  best  match  to  test  data. 

J(T,A)  =  matching  error  obtained  from  test  data,  using  the  average 
parameter  set. 

Finally,  we  compute  the  following  "matching  error  ratios": 
J  (B,A)/J  (B,B)  ;  J (T,A)/J (T,T) ;  and,  if  we  wish  to  reduce  the 
results  to  a  single  number,  the  average  of  these  two  error  ratios. 
In  a  qualitative  sense,  the  greater  the  matching  error  ratios,  the 
more  significant  are  the  differences  between  the  parameters 
identified  for  the  baseline  and  test  conditions.  In  addition  to 
providing  a  collective  test  of  the  entire  parameter  set,  this 
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scheme  may  also  be  useful  to  test  a  single  parameter  or  a  subset  of 
parameters . 

As  shown  by  Levison  [7]  ,  a  good  approximation  to  the  joint 
match  to  multiple  data  sets  can  sometimes  be  obtained  by  simply 
matching  the  average  data.  Thus,  to  obtain  the  "average  parameter 
set",  one  would  first  obtain  a  point-by-point  ensemble  average  of 
the  (reduced)  baseline  and  test  data,  and  then  identify  parameters 
to  match  the  average  data  set.  This  procedure  is  valid  if  the  same 
task  description  applies  to  the  two  experimental  conditions;  i.e., 
if  both  tasks  can  be  modeled  identically  except  for  quantitative 
differences  in  pilot-related  parameters.  Experiments  designed  to 
explore  training  effects,  environmental  stress,  or  interference 
from  other  concurrent  tasks  often  meet  this  restriction. 

The  identification  scheme  reviewed  here  —  including  the 
qualitative  cross-comparison  method  of  significance  testing  —  has 
been  recently  utilized  to  explore  the  effects  of  training  on 
pilot-related  model  parameters  [7,8]. 

CONCLDS IONS 

A  variety  of  methods  exist  for  identifying  pilot  response 
characteristics  in  manual  control  tasks.  Theoretical  analysis  and 
practical  experience  indicate  that  pilot  response  can  be  most 
accurately  identified  when  the  following  conditions  obtain:  (a) 
steady-state  task  environment,  (b)  ability  to  measure  the  external 
forcing  function(s),  and  (c)  construction  of  the  tracking  input(s) 
as  a  sum  of  sinusoids.  Pilot-related  parameters  of  the  optimal 
control  model  for  the  human  operator  can  then  be  identified  from 
time-domain  and  frequency-response  measures  computed  from 
experimental  time  histories. 
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The  Advanced  Fighter  Technology  {AETT)/F-16  program  is  in  response  to 
today's  European  scenario,  characterized  by  increased  numbers  of  enemy  targets 
both  on  the  ground  and  in  the  air  and  an  increasingly  hostile  air  space  sur¬ 
rounding  these  targets.  This  changing  environment  requires  timely  improvements 
in  present  USAF  fighter  lethality  and  survivability.  The  primary  and  continuing 
objective  of  the  AFIT  program,  co-sponsored  by  the  Air  Force,  NASA  and  Navy,  is 
to  provide  for  the  development,  integration,  flight  evaluation,  and  demonstration 
of  emerging  fighter  technologies,  and  transition  of  the  integrated  technologies 
to  future  system  applications.  The  AFIT  Fighter  Attack  Technology  (AFTI/F-16) 
program  will  develop,  integrate  and  flight  test  a  set  of  technologies  to  improve 
the  survivability  and  weapon  delivery  accuracy  of  tactical  fighters  in  air-to-air 
and  air-to-ground  attacks,  through  integration  of  advanced  technologies  into  a 
single  seat  demonstrator  vehicle  which  permits  a  realistic  evaluation  of  technology 
benefits,  penalties  and  overall  mission  effectiveness. 

The  AFTI/F-16  vehicle  (depicted  in  Figure  1)  has  particular  importance  as 
a  long  life  demonstrator  aircraft  with  the  flexibility,  versatility,  and  capability 
in  terms  of  performance  and  systems  to  serve  as  a  future  technology  development 
testbed.  A  full-scale  development  F-16  aircraft  is  the  test  vehicle.  Extensive 
modifications  were  made  for  installation  of  a  sophisticated  data  instrumentation 
system,  modified  inlet  with  canards,  new  flight  control  system,  and  a  dorsal  fairing 
to  accommodate  the  instrumentation  equipment.  Additional  information  on  the 
AFTI/F-16  can  be  found  in  Reference  1. 

The  overall  objective  of  the  AFIT/F-16  Advanced  Development  Program  is  to 
demonstrate  separately,  and  in  combination,  advanced  fighter  technologies  to 
improve  air-to-air  (AA)  and  air-to-surface  (AS)  weapon  delivery  accuracy  and 
survivability.  These  technologies  include  a  Digital  Flight  Control  System  (DFCS) , 
Automated  Maneuvering  Attack  System  (AMAS) ,  pilot/vehicle  interface  (PVI)  advance¬ 
ments,  and  advanced  task-tailored  control  modes  utilizing  direct  force  control  and 
weapon  line  pointing.  Development,  integration,  and  flight  validation  of  these 
fighter  attack  technologies  have  been  separated  into  DFCS  and  AMAS  program  phases. 

The  DFCS  is  a  full-authority,  triplex,  i  digital  fly-by-wire  flight  control 
system.  The  DFCS  is  mechanized  to  implement  task- tailored  manual  control  modes, 
including  decoupled  (six  independent  degrees  of  freedom  or  control-configured 
vehicle)  flight  control.  Figure  2  shows  that  the  pilot  need  only  push  a  button 
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to  change  the  functions  of  cockpit  controllers  and  displays.  For  the  AMAS 
phase,  the  effective  utilization  of  the  advanced  technologies  requires  the 
integration  (coupling)  of  the  fire  and  flight  control  functions .  The  integrated 
system  will  tie  together  a  director  fire  control  system,  an  advanced  sensor- 
tracker,  and  the  flight  control  syston  to  provide  precise  autcmated  weapon  line 
control  and  weapons  delivery.  With  the  coupled  system  the  azimuth  and  elevation 
fuselage  pointing  capability  of  the  aircraft  provides  an  expanded  envelope  of 
fire  control  solutions;  i.e. ,  an  enlarged  pipper.  The  pilot  need  only  capture 
the  target  within  the  expanded  pipper  envelope  and  the  fire  control  system 
will  automatically  cattnand  the  flight  control  sy  start  to  null  aiming  errors  to 
assure  a  hit.  This  concept  will  profoundly  influence  fighter  effectiveness  in 
both  AA  and  AS  missions. 

Pilot/vehicle  interface  advancanents  will  be  incorporated  to  provide  crew 
station  capabilities  and  environment  ccrrmensurate  with  the  increase  in  total 
vehicle  capabilities  provided  by  the  other  technologies  in  each  phase.  The 
DPCS  phase  will  focus  on  core  technology  development.  The  technologies  of 
prime  interest  will  be  manual  flight  path  control,  avionics  integration,  and 
advanced  controllers  and  displays.  In  the  AMAS  phase  the  allocation  of  function 
between  the  pilot  and  vehicle  will  be  redistributed  as  a  result  of  the  DFCS 
experience.  Those  tasks  best  performed  by  the  machine  will  be  automated.  Tech¬ 
nological  advances  in  sensors,  fire  control  modes,  and  weapons  fusing  will  be 
integrated  with  the  DFCS  capabilities.  Figure  3  shows  a  rough  schedule  of  the 
AFTI/F-16  program  and  the  relative  time  frame  where  technological  transition  will 
occur  from  DFCS  to  AMAS,  and  from  the  AFTI/F-16  program  to  current  and  future 
fighter  aircraft. 

An  example  of  advanced  technology  integration  and  utilization  is  in  the 
AMAS  precision  low  altitude  maneuvering  attack  scenario.  The  technologies 
involved  in  this  scenario  include: 

1.  Flight  path  control  with  full  authority  digital  flight  control. 

2.  Task  automation  with  integrated  flight  and  fire  control  and  low 
altitude  radar  autopilot. 

3.  Advanced  sensor-tracker  with  low  drag  FLIR  and  laser  ranger 
installation. 

4.  Integrated  avionics  and  weapons  fusing. 

5.  Cockpit  development  including  multi-purpose  displays,  wide  field  of 
view  heads  up  display,  helmet-mounted  sight  and  voice  command. 

6.  Weapons  interface  with  pilot  consent  and  auto-release. 

These  technologies  together  give  the  AFTI/F-16  the  ability  to  more  effect¬ 
ively  attack  ground  targets  as  depicted  in  Figure  4.  A  low  altitude  radar 
autopilot  allows  survivable  ingress  and  egress.  AMAS  autcmated  air-to-surface 
bombing  nodes  provide  the  capability  for  flexible  target  acquisition,  precise 
tracking,  automated  ingress/attack  steering,  and  automated  weapon  release  for 
both  low  altitude,  or  standoff  delivery  direct,  or  high-g  turning  attacks. 
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Target  acquisition  of  either  preplanned  or  in-flight  designated  targets  can 
be  accomplished  through  use  of  the  helmet-mounted  sight  (HMS)  for  wide  field 
of  coverage  visual  designations,  the  heads  up  display  (HUD)  for  narrow  field 
of  coverage  visual  designations,  or  the  radar  ground  map  for  night/all  weather, 
standoff  designations.  Once  target  acquisition  has  been  obtained  the  AFTI 
sensor/tracker  with  imaging  FLIR  and  laser  ranger  can  be  used  for  track,  ID,  and 
precise  position  measurement  of  the  target.  The  pilot  can  jink  manually  to  avoid 
threats  until  engaging  an  automated  attack  mode  which 'may  follow,  at  the  pilot's 
option ,  either  a  conventional  wings-level  or  high-g  turning  weapon  delivery 
trajectory  with  automatically  computed  weapon  release.  In  this  scenario  the 
AFTI/F-16  frees  the  pilot  to  become  a  true  weapons  system  manager,  concentrating 
on  target  acquisition,  attack  planning  and  threat  avoidance.  Figure  5  shows  that 
this  attack  profile  is  clearly  more  survivable  than  that  of  the  basic  F-16A.  The 
AMAS  profile  is  expected  to  give  an  order  of  magnitude  increase  in  survivability 
without  degrading  weapon  delivery  accuracy. 

PILOT-VEHICLE  INTERFACE  EVALUATION 


The  evaluation  of  pilot- vehicle  interface  (PVI)  during  the  AFTI/F16 
development  and  flight  test  program  will  indeed  be  challenging.  The 
opportunity  is  there  to  tie  together  existing  knowledge  in  several  disciplines: 
flying  qualities,  pilot  dynamics,  pilot  workload,  automation,  and  mission 
effectiveness.  During  the  first  year  of  flight  testing  the  emphasis  will  be 
on  manual  control  of  an  aircraft  having  eight  different  task-tailored  flight 
control  modes.  Advances  in  our  understanding  of  flying  qualities  and  pilot 
dynamics  could  be  made.  Also,  new  measures  of  pilot  workload  and  mission 
effectiveness  will  be  checked  out.  In  the  second  year  of  flight  testing  the 
enphasis  will  shift  to  automated  control  of  the  aircraft  to  improve  mission 
effectiveness  and  reduce  pilot  workload.  Criteria  in  those  areas  are  expected 
to  be  explored,  validated  and  applied. 

Presently,  planning  for  the  first  year  of  flight  testing  is  virtually 
complete.  Several  pilot-in-the-loop  (closed  loop)  maneuvers  are  contained 
in  the  test  plan.  They  are  drawn  from  various  test  activities  in  support  of 
different  engineering  disciplines,  but  are  listed  together  in  Table  1.  Each 
run  enumerated  in  the  table  corresponds  to  a  particular  test  configuration 
and  flight  condition  called  out  in  the  flight  test  plan.  In  general,  the 
AFTI/F-16  vehicle  will  respond  differently  for  each  of  these  runs.  The 
closed- loop  maneuvers  will  be  used,  from  the  pilot- vehicle  interface  view¬ 
point,  to  (1)  determine  the  closed  loop  dynamic  characteristics  of  the  AFTI/F-16 
vehicle,  (2)  generate  pilot  subjective  ratings  and  comments  while  performing 
various  tasks,  (3)  quantitatively  measure  workload,  and  (4)  measure  task  per¬ 
formance  (mission  effectiveness) .  Pilot  technique  or  pilot  dynamics  may  also 
be  studied.  This  wealth  of  flight  test  data  will  allow,  within  available 
resources,  a  broad  assessment  of  various  criteria  for  flying  qualities,  work¬ 
load,  and  mission  pierf ormance .  Taken  as  a  whole,  this  broad  assessment  offers 
a  unique  opportunity  to  increase  our  understanding  of  pilot-vehicle  interface 
(PVI)  and  how  best  to  evaluate  PVI  in  flight. 
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Flying  Qualities 


A  major  portion  of  the  first  year  of  the  AFTI/F-16  flight  test  program 
will  be  the  flying  qualities  evaluation  of  the  manual  control  modes.  There 
are  in  essence  eight  different  AFTI/F-16  airplanes  corresponding  to  the  eight 
task-tailored  manual  control  modes,  each  of  which  must  demonstrate  satisfactory 
flying  qualities.  Many  of  these  control  modes  are  unconventional  (decoupled  force 
or  pointing) .  New  flying  qualities  criteria  for  such  modes  have  been  explored 
(see  Reference  2  where  a  simple  frequency-response  criterion  is  evaluated) ,  but  in 
general  the  latest  flying  qualities  specifications  lack  flight-test  validated 
evaluation  criteria.  The  opportunity  exists  in  the  AFTI/F-16  flight  test  program 
to  explore  new  flying  qualities  criteria  in  a  mission- relevant  context.  The 
following  paragraphs  will  describe  how  this  may  be  done  using  the  state-of-the-art 
in  flight  test  techniques,  instrumentation,  and  data  analysis  capabilities. 

The  time  responses  of  the  AFTI/F-16  vehicle  to  various  cockpit  controller 
inputs  will  be  compared  with  predictions  using  "canned"  maneuvers  (such  as 
doublets)  where  the  pilot  is  not  performing  any  real  task  with  the  vehicle. 

These  response  time  histories  will  be  used  to  help  validate  simulations  of  the 
AFTI/F-16  which  then  may  be  used  to  generate  analytical  flying  qualities  data  to 
compare  against  suitable  criteria.  However,  pilot-in-the-loop  (closed  loop) 
maneuvers  listed  in  Table  1  must  be  relied  upon  to  provide  the  pilot  a  task  to 
perform  so  that  he  can  give  a  valid  pilot  rating  and  good  pilot  comments  (see 
Reference  3) .  The  details  of  these  closed  loop  maneuvers  are  contained  in  the 
appendixes.  Broadly,  the  PVI  evaluation  maneuvers  can  be  grouped,  relative  to 
their  use  in  the  evaluation  of  flying  qualities,  as  follows: 

a.  Frequency  Response  Analysis  Maneuvers.  These  are  structured  to  excite 
the  motion  of  the  vehicle  so  that  a  fast-fourier  transform  of  the  resulting  time 
histories  will  yield  a  good  frequency-domain  representation  of  the  aircraft 
response  to  pilot  control  inputs.  Flying  qualities  analysis  of  aircraft 
frequency  response  can  then  be  made. .  These  maneuvers  provide  the  pilot  with  a 
realistic  tracking  task  to  perform  although  he  is  asked  to  increase  his  concentra¬ 
tion  and  gain  to  a  level  that  artificially  excites  the  motion  of  the  aircraft. 
Tracking  performance,  read  from  gun  camera  film,  has  in  the  past  not  correlated 
with  pilot  ratings.  This  is  because,  in  pilot  dynamics  terminology,  the  pilot 
has  increased  his  remnant  (noise) .  However ,  good  pilot  ratings  and  comments  are 
generated.  These  maneuvers  include  handling  qualities  during  (air-to-air) 
tracking  (see  also  Reference  4)  and  air-to-surface  gun  and  bomb  tracking. 

b.  Artificial  Test  Maneuvers.  In  a  sense  all  test  maneuvers  are  artificial, 
but  the  weapons  separation  maneuvers  are  exceptional.  These  maneuvers,  described 
in  Appendix  B,  will  tax  the  pilot's  ability  to  attain  specific  flight  conditions 
and  unusual  attitudes  within  precise  tolerance  goals  over  a  predetermined  weapons 
release  point.  Flying  qualities,  pilot  dynamics,  and  workload  studies  will 
benefit  from  these  artificially  intense  maneuvers. 

c.  Continuous  Task  Maneuvers.  Formation  flying,  refueling,  landing 
(Appendix  A) ,  and  flying  in  turbulence  (Appendix  C)  are  real  mission  tasks  that 
will  demonstrate  the  flying  qualities  of  the  AFTI/F-16  vehicle.  Since  they  are 
continuous  (in  terms  of  constant  flight  condition  and  pilot  technique)  these 
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maneuvers  would  be  perfect  for  evaluation  of  pilot  dynamics  if  a  crucial 
limitation  can  be  overcome.  This  is  the  difficulty  of  measuring  the  actual 
ccmmand  signal  during  flight  to  which  the  pilot  is  responding.  For  example, 
how  can  we  measure  distance  from  the  lead  airplane  (formation  flying)  or  the 
distance  to  the  refueling  boom?  In  seme  cases  the  ccmmand  signal  can  be  picked 
off  the  heads-up  display  or  otherwise  inferred. 

d.  Evaluation  Maneuvers  (Appendix  D) .  These  are  intended  to  duplicate 
real  combat  scenarios.  Each  project  pilot  will  be  allowed  to  use  his  own  ■ 
technique  based  on  his  experience  with  the  unique  task-tailored  flight  control 
modes  and  standard  F-16  fire  control  system.  Flying  qualities  criteria 
developed  in  previous  tests  or  analysis  can  be  related  to  mission  effectiveness 
and  pilot  workload  during  these  maneuvers  both  subjectively  and  quantitatively. 

The  last  two  sections  of  this  paper  will  describe  hew  that  may  be  done.  Pilot 
dynamics  measured  during  these  discrete  operational  maneuvers  may  also  provide 
insight  into  how  different  pilots  perform  real  combat  tasks.  However,  pilot 
training  and  experience  will  be  limited  by  constraints  in  simulator  and  flight 
time.  This  is  a  limitation  in  any  advanced  development  program. 

Instrumentation  on  the  AFTI/F-16  is  described  in  Appendix  E.  The  vehicle 
.will  be  heavily  instrumented  for  many  different  engineering  disciplines. 

A  partial  list  of  instrumentation  parameters  to  be  measured  during  flying 
qualities  flight  tests  is  given  in  Table  2.  Data  processing  will  be  accomplished 
on  both  a  real-time  and  post-flight  basis.  Since  the  projected  flight  rate  of  the 
AFTI/F-16  will  be  three  flights  per  week,  f light- to-f light  decisions  will  be 
based  on  real-time  engineering  units  strip  charts,  crossplots,  and  tabulations 
displayed  in  the  NASA  control  room.  Post-flight  data  processing  consists  of 
more  extensive  second-generation  plots  and  listings  as  well  as  third  generation 
processing  performed  by  AFFTC  engineers.  Third-generation  programs  will  be 
used  for  fast  fourier  transform  analyses  (FRA;  see  Reference  5) ,  parameter 
identification  (MMLE3;  see  Reference  6),  and  as  prediction  and  standardization 
routines  (EASY5;  see  Reference  7) . 

Pilot  Workload 


During  the  first  year  of  the  AFTI/F-16  flight  test  program  physiological 
data  will  be  recorded  and  analyzed  to  help  answer  questions  about  the  pilot 
workload  associated  with  advanced  fighter  aircraft  technologies.  Various 
subjective  rating  techniques  will  supplement  physiological  workload  analyses. 

Two  new  subjective  techniques  to  be  used  on  the  AFTI/F-16  project,  System 
Operability  Maasurenent  Algorithm  (SCMA)  and  Subjective  Workload  Assessment 
Technique  (SWAT)  are  very  adequately  covered  in  References  8  and  9  in  this 
proceedings.  The  recording  and  analysis  of  pilot  physiological  data  will  be 
performed  in  collaboration  with  the  Air  Force  School  of  Aerospace  Medicine 
(AESAM) ,  Air  Force  Flight  Test  Center  (AFFTC)  and  NASA.  For  details  on  the 
development  of  this  physiological  measurement  technique  see  Reference  10  in 
this  proceedings.  EKG  is  the  primary  physiological  parameter  to  be  measured 
during  this  phase.  From  the  EKG  data,  heart  rate  (an  variance)  and  R-wave 
amplitude  (and  variance)  will  provide  insights  into  pilot  workload.  Respiration 
rate  is  also  being  investigated  as  an  additional  measurement  parameter. 

Figure  6  illustrates  the  development  of  the  physiological  workload  measure¬ 
ment  techniques.  Flights  with  physiological  monitoring  will  begin  one  month 
after  first  flight.  Months  two  through  four  will  concentrate  on  developing 
procedure,  shaking  out  equipment,  developing  workload  criteria  and  cross-validating 
physiological  data  collected  earlier 'during  simulator  and  centrifuge  experiments 
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(see  References  10,  11  and  12) .  Months  four  through  eight  will  consist  of 
answering  "simple"  questions  about  the  workload  associated  with  technologies 
in  a  mission  context  (e.g.  does  a  decoupled  mode  cause  less  pilot  workload 
during  fine  tracking  than  the  normal  mode?).  Months  8  through  12  will  involve 
answering  complex  issues  about  technologies,  making  recommendations  about 
allocation  of  function  between  pilot  and  machine,  in  terms  of  automation,  for 
Phase  II,  and  reporting  findings. 

Both  micro  and  macro  analyses  of  pilot  workload  will  be  conducted  on 
flights  of  interest.  Micro  analyses  will  be  made  on  selected  maneuvers  to 
determine  the  workload  associated  with  a  given  technology  within  a  given 
maneuver  or  mission  segment.  Maneuvers  of  high  interest  are  listed  in  Table  1. 
Also,  micro  workload  analysis  will  be  made  on  an  exception  basis,  i.e.  where  an 
unexpectedly  high  workload  situation  occurs.  Macro  analysis  will  be  made  to 
determine  the  fatigue  incurred  during  that  total  flight. 

Due  to  the  small  number  of  pilots  (6)  and  the  effects  of  learning  in  a 
complex  aircraft,  there  are  seme  minimum  criteria  for  examining  maneuvers  of 
interest  for  statistical  comparisons: 

1.  All  pilots  must  fly  the  maneuver. 

2.  Each  pilot  repeats  the  maneuver  three  times. 

3.  Learning  effects  are  stabilized  or  at  least  identified. 

Table  3  illustrates  the  interaction  of  advanced  technologies  and  flight 
test  activities.  Advanced  technologies  are  advertised  to  enhance  the  pilot's 
ability  to  manage  aircraft  systems  (i.e.  control  of  aircraft,  weapons  systen 
management  and  support  systems  management) .  Flight  test  activities  include 
flying  qualities,  weapons  separation  and  DFCS  evaluations .  Maneuvers  within 
each  flight  test  activity  are  listed  in  the  appendixes.  The  desirability  or 
goodness  of  advanced  technologies  will  be  determined  by  analyzing  pilot  workload 
and  task  performance  for  the  appropriate  flight  test  activity. 
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MISSION  EFFECTIVENESS  EVALUATION 


Complementary  to  the  evaluation  of  Pilot-Vehicle  Interface  (workload 
or  flying  qualities)  is  the  measurement  of  task  performance.  Presently, 
few  attaripts  have  been  made  in  flight  to  quantitatively  measure  and  re¬ 
late  pilot  workload  to  task  performance  during  the  same  maneuver.  Most 
of  these  attempts  were  helicopter  flight  tests  (e.g.  References  13  and 
14) .  Fixed  wing  experience  at  AFFTC  so  far  (Reference  15)  is  that  task 
performance  measures  (i.e.  rms  pipper  error  statistics)  recorded  during 
Handling  Qualities  During  Tracking  Maneuvers  do  not  correlate  with  air¬ 
plane  handling  qualities  as  assessed  by  pilots.  It  is  clear  that  better 
ways  to  measure  and  explain  task  performance  are  necessary  to  adequately 
evaluate  Pilot-Vehicle  Interface  during  the  AFIT/F-16  flight  test  program. 

Mission  effectiveness  will  be  the  bottom  -  line  measure  of  task  performance 
during  the  AFTI/F-16  program.  The  tasks,  in  this  case,  represent  the  two 
tasks  inherent  in  any  combat  mission:  killing  the  target  and  surviving 
enemy  threats.  Mission  effectiveness  analysis  then  will  quantify  these  two 
bottom  -  line  measures  of  combat  mission  effectiveness  achieved  by  the  pilot 
and  aircraft  on  a  given  combat  maneuver  as: 

a.  Weapon  delivery  accuracy  (probability  of  killing  air  or  surface 
targets  with  certain  weapons)  and 

b.  Survivability  (probability  of  surviving  certain  ground-based 
defenses) . 

These  are  expected  to  provide  a  meaningful  comparison  of  the  relative  use¬ 
fulness  of  the  AFTI/F-16  technologies  to  perform  tine  intended  combat  missions. 
Since  the  two  tasks  of  killing  the  target  and  surviving  are  often  in  conflict 
with  each  other,  they  will  demand  the  pilot's  total  attention  and  effort  and  so 
should  provide  a  sound  basis  for  validating  in-flight  measures  of  workload  as 
well  as  for  demonstrating  the  mission  relevance  of  advanced  flying  qualities 
criteria.  New  insight  into  what  may  be  the  primary  contributors  to  weapon  de¬ 
livery  accuracy  and  survivability  measures  will  also  be  gained  for  each  mission 
segment  so  that  a  preliminary  assessment  of  the  feasibility  of  a  performance  or 
workload  specification  may  be  made. 

MISSION  EFFECTIVENESS  ANALYSIS  The  basic  approach  to  mission  effectiveness 
analysis  on  the  AFTI/F-16  program  is  to  use  the  General  Dynamics  simulator 
as  the  primary  source  of  quantitative  mission  effectiveness  data  for  meeting 
contractual  mission  effectiveness  specifications. 

Simulations  are  planned  to  compare  the  DFCS-phase  AFTI/F-16  with  the  baseline 
F-16A  aircraft  and  with  the  fully-integrated  AMAS-phase  AFTI/F-16  using  common 
mission  scenarios.  Proposed  data  collection  and  analysis  details  for  the  DFCS 
Mission  Effectiveness  Simulations  are  contained  in  Appendix  F.  Flight  testing 
will  be  done  to  verify  simulation  results  using  the  proposed  DFCS  evaluation 
maneuvers  described  in  Appendix  D.  Flight  test  and  simulator  data  will  be 
processed  through  the  same  mission  effectiveness  computer  programs  by  General 
Dynamics  to  provide  this  verification.  Subjective  pilot  opinion  and  physio¬ 
logical  wokload  data  will  also  be  used  to  independently  validate  simulation 
fidelity. 
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COMPARISON  WITH  PILOT  WORKLOAD  We  expect  that  evaluating  performance  in  two 
primary  tasks  (hitting  the  target  and  surviving)  should  allow  a  good  comparison 
with  physiological  and  subjective  measures  of  workload  because  the  two  tasks 
normally  saturate  the  pilot's  cognitive  abilities.  Success  by  others  with  sec- 
condary  task  measures  of  workload  also  gives  us  confidence  that  dual  mission 
effectiveness  measures  will  relate  with  pilot  workload.  For  each  flight  test 
maneuver  the  pilot  will  need  to  be  properly  instructed  and  motivated  as  if  in 
canbat,  that  is,  to  be  as  elusive  as  possible  for  survival  while  placing  the 
pipper  on  the  target  long  enough  to  assure  a  kill.  After  each  flight,  subjective 
pilot  ratings  and  camients  will  be  taken.  Flight  data  will  be  processed  by  AFFTC, 
AFSAM,  and  General  Dynamics  and  compared.  Each  pilot's  individual  physiological 
variability  using  a  "control"  maneuver  (e.g.  takeoff) .  These  comparisons  will 
not  only  validate  workload  measures  but  may  also  lead  us  to  prcmisinig  mission 
effectiveness  criteria  that  may  eventually  be  used  as  a  perfonrence  or  workload 
specification.  The  following  steps  show  how  this  may  take  place: 

1.  Establish  realisitc  mission  scenarios,  initial  conditions  and 
measurement  and  data  reduction  procedures  for  collecting  the 
necessary  data.  Determine  utility  simulation  versus  flight  test. 

2.  Develop  ways  to  normalize  data  for  differences  among  pilots  and 
their  various  skill  levels  and  thecniques. 

3.  Relate  flying  qualities  criteria  to  workload  and  mission  effectiveness 
criteria.  Recommend  profitable  avenues  of  research. 

a.  DFCS-Phase  of  AFTI/F-16  program  emphasizes  manual  control  of 
flight  where  flying  qualities  criteria  and  specifications 
apply.  Levels  of  flying  qualities  will  be  related  to  workload 
and  mission  effectivensss  criteria. 

b.  AMAS-Phase  of  AFTI/F-16  emphasizes  automated  control  of  flight 
where  workload  and  mission  effectivensss  criteria  (developed  in 
the  previous  phase)  apply  instead  of  flying  qualities.  Candidate 
specification  criteria  for  workload  or  mission  effectiveness  will 
be  recommended  for  futher  research. 

MISSION  EFFECTIVENESS  CRITERIA  Calculations  of  the  probability  of  target 
kill  or  probability  of  survival  require  specialized  computer  programs  (as  in 
Reference  17) .  A  simpler  approach  would  be  to  develop  mission  effectiveness 
criteria  which  could  be  measured  directly  during  flight  test.  Such  criteria 
would  need  to  be  proven  to  be  components  of  either  probability  of  kill  or 
survival.  The  AFTI/F-16  program  will  provide  an  opportunity  to  examine  the 
components  of  the  two  mission  effectiveness  measure,  accuracy  and  survivability 
for  various  mission  tasks  as  described  below. 

1-  Accuracy.  Listed  in  Table  4  are  same  of  the  primary  contributors  to 
hitting  the  target  successfully,  assuming  constant  weapon  character¬ 
istics.  The  primary  contributor  ought  to  be  same  measure  of  fine 
tracking  ability  which  efficiently  allows  the  pilot  to  attain  pre¬ 
cise  weapon  release  conditions.  A  secondary  contributor  to  accuracy 
is  same  measre  of  gross  acquisition  ability.  These  hypotheses  will 
be  validated  through  correlation  of  flight  data  with  probability  of 
target  kill  calculations . 

2.  Survivablity.  For  cents tant  threat  characterisitic,  probability  of 
survival  should  be  proportional  to  same  measure  of  elusiveness. 

Possible  measures  of  elusivenss  are  listed  in  Table  5.  These  primarily 
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apply  to  ground-based  threats.  For  threats  from  enany  aircraft  a  primary 
contributor  to  vulnerability  is  loss  of  energy  (airspeed  or  altitude) . 
Secondary  contributors  to  survivability  are  the  primary  contributors  to 
weapon  delivery  accuracy  since  the  ability  to  quickly  kill  the  target 
reduces  the  time  exposed  to  hostile  enemy  action. 

SUMMARY 


The  AFTI/F-16  flight  test  program  will  offer  many  opportunities  for 
the  Air  Force,  NASA,  and  Navy  to  evaluate  pilot- vehicle  interface  and 
mission  ef fectivenss .  The  AFTI/F-16  vehicle  contains  an  advanced  systems 
capability  that  will  enhance  the  pilots  ability  to  control  the  aircraft. 
Automated  modes  will  configure  these  systems  to  help  the  pilot  and  vehicle 
to  better  perform  realistic  combat  mission  tasks.  The  first  year's  flight 
test  plan  has  many  pilot- in-the- loop  maneuvers  that  will  permit  the  eval¬ 
uation  of  flying  qualities,  pilot  workload,  weapon  delivery  accuracy,  and 
survivability.  New  criteria  are  expected  to  be  developed  or  validated  in 
each  of  these  areas.  Tire  results,  if  adequate  resources  are  applied,  should 
advance  our  capability  to  test  and  evaluate  pilot-vehicle  interface  and 
mission  effectiveness. 
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TABLE  1 


PVI  EVALUATION  MANEUVERS 


DEVELOPMENT  MANEUVERS  NO.  RUNS 

HANDLING  QUALITIES  DURING  TRACKING  21 

AIR-TO-SURFACE  GUN  TRACKING  6 

AIR-TO-SURFACE  BOMB  TRACKING  10 

FORMATION  FLYING  A 

AERIAL  REFUELING  9 

ILS  APPROACH  TO  LANDING  2 

RESPONSE  TO  TURBULENCE  10 

WEAPONS  SEPARATION  27 

EVALUATION  MNEjjVERS  NO.  RUNS 


AIR-TO-SURFACE  BOMBING  TECHNIQUES 

CONTINUOUSLY  COMPUTED  IMPACT  POINT 
DIVE  TOSS 

CONTINUOUSLY  COMPUTED  RELEASE  POINT 
LOW  ALTITUDE  DROGUE  DELIVERY 
AIR-TO-SURFACE  STRAFE 
AIR-TO-AIR  GUNNERY 
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TABLE  2 


FLYING  QUALITIES  INSTRUMENTATION  LIST 

Instrumentation  parameters  to  be  used  during  the  Flying 
Qualities  and  DFCS  Develocment  flight  tests  are  as  follows: 


PARAMETER 

Sideslip  -  Delta  Pressure 
RAM  Air  Temp 
Landing  Gear  Handle  Pos . 
Landing  Gear  Handle  Pos . 

Alt.  Flap  Switch 
Alt.  Flap  Switch 
Air  Refuel  Door  Pos. 

Air  Refuel  Door  Pos. 

IFFC  Switch  Position 
MLG  Weight-On-Wheels 
Spin  Chute  Deploy 
Spin  Chute  Jettison 
Dogfight  Switch 
Missile  0 ' Ride  Switch 
Weapon  'Release  Button 
Weapon  Release 
Trigger-lst  Detent 
Trigger-2nd  Detent 
AO A  >  29° 

Stall  Warning 

CADC  Good 

INS  Good 

Drag  Modulation 

Auto  Stores 

Norm.  Accel  -  FLCC 

Norm.  Accel.  -  CG 

Norm.  Accel  -  FLCC  Sel. 

Lat  Accel. 

Lat .  Accel  -  FLCC  Sel. 

Long.  Accel.  -  FLCC 

Pitch  Angel 

Roll  Angle 

Yaw  Angle 

Roll  Rate 

Pitch  Rate 

Yaw  Rate 

Forward  Fuel  Qty. 

AFt .  Fuel  Qty. 

L/H  Wing  Fuel  Qty. 

R/H  Wing  Fuel  Qty. 

Basic  Fuel  Flow 

Total  Fuel  Flow 

Fuel  Temperature 

LEF  Hinge  Mom.  BL  62.15 

LEF  Hince  Mom!  BL  96.68 

LEF  Hinge  Mem.  BL  125.1 

LEF  Hinge  Mom.  BL  145.4 

L/H  Flaperon  KM  (hinge  Momen 

R/H  Flaperon  KM 

L/H  Horizontal  Tail  KM 

R/H  Horizontal  Tail  KM 

L/H  Canard  HM 


PARAMETER 

Mach  Number 

Altitude 

Altitude 

KCAS 

KTAS 

Ps 

qc 

Angle  of  Attack  -  NB 

Angle  of  Attack  -  FLCC 

L/K  AOA  Cone 

R/H  AOA  Cone 

AOA  Side  Mount 

Sideslip  Angle  -  NB 

Sideslip  Angle  -  FLCC 

Sideslip  -  Lower  Probe 

R/H  Canard  HM 

Rudder  HM 

Nl-RPM 

N2-RPM 

Nozzle  Area 

L/H  Wheel  RPM 

R/H  Wheel  RPM 

Spin  Chute  Load 
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DFCS  PVI  EVALUATION  FACTORS 
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TABLE  4 


ACCURACY  MEASURES 

PHir  PKILL  FOR  CONSTANT  WEAPON  CHARACTERISTICS 

PRIMARY  CONTRIBUTOR  IS  FINE  TRACKING  ACCURACY 
o  PIPPER  ERRORS  AT  WEAPON  RELEASE 
o  TIME  ON  TARGET 

o  RMS  TRACKING  ERROR 

o  FLYING  QUALITIES  PARAMETERS 

SECONDARY  CONTRIBUTOR  IS  GROSS  ACQUISITION 
o  TIME  TO  STABILIZE  PIPPER 

o  AERODYNAMIC  CAPABILITY 

TABLE  5 

SURVIVABILITY  MEASURES 
Ps  FOR  CONSTANT  THREAT  CHARACTERISTICS 

PRIMARY  CONTRIBUTOR  IS  ELUSIVENESS 
o  ALTITUDE  PROFILE 

o  EXPOSURE  TIME 

o  JINKING  MANEUVERS 

o  GROSS  ACQUISITION  TIME 

o  STANDOFF  RANGE 

o  (FOR  AIR-TO-AIR)  LOSS  OF  ENERGY 

SECONDARY  CONTRIBUTOR  IS  FINE  TRACKING  ACCURACY 

o  FINE  TRACKING  TIME 

o  FLYING  QUALITIES  PARAMETERS 
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DECOUPLED  {  DECOUPLED  |  DECOUPLED  DECOUPLED 

Sp-FPME*  <>  Sp-FPME  0  Sp-PRME  *  $  Sp-PRME  4 


SR-Roll  Rate  o  SR-Roll  Rate  o  Sr-RoII  Rate  o  SR-Roll  Rate  o 

P-Translation  P-Flat  Turn  □  P-Pointing  B  P-Pointing  B 

T-Translation  T-DiRECT  LIFT  T- Pointing  y  T-Pointing  y 


NOTE:  Mode  control  axes  with  sane  symbols  have  identical  control 

laws  and  gains 

•  FPME  -  FLIGHT  PATH  MANEUVER  ENHANCEMENT 
PR  ME  -  PITCH  RATE  MANEUVER  ENHANCEMENT 

,©  Identical  outside  bombing  envelope  only 


Figure  2  AFTI/F-16  Multinode  Flight  Controller  Commands 
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Figure  3  A  FT  l/F-1 6  SCHEDULE 


Figure  4 

AUTOMATED  MANEUVERING  ATTACK 
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PILOT  ENGAGES  SYSTEM 
(Ingress  Streering) 


Figure  5  SURVIVABILITY  COMPARISON  PROFILES 
FOR  AIR-TQ-SURFACE  BOMBING 
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AUTO  PILOT  AT 
300  FT  &  500  KTS 


TEST  ACTIVITIES  #  FLTS 
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APPENDIX  A 


CLOSED  LOOP  FLYING  QUALITIES  MANEUVERS 


Formation 


1.  The  evaluation  of  flying  qualities  during  formation 
will  be  conducted  in  conjunction  with  other  tests  and 
by  flying  formation  with  chase  aircraft  while  enroute 
to  and  from  the  test  ranges . 

2 .  Evaluate  formation  flying  in  the  flight  control  modes 
specified  in  Table  8.2-2.  Evaluate  the  flight  control 
laws  and  gains  implemented  in  each  specified  mode 
while  flying  a  formation  closed  loop  task. 

3.  Detailed  procedures  and  conditions  are  not  specified 
so  as  to  encourage  an  unbiased  evaluation  of  the 
utility  of  the  various  DFCS  modes  . 

Aerial  Refueli.nc 


1.  Evaluate  aircraft  flying  qualities  during  aerial 
refueling  by  utilizing  standard  aerial  refueling 
procedures . 

2 .  The  flying  qualities  during  refueling  will  be 
evaluated  in  the  flight  control  modes  specified. 
Evaluate  the  flight  control  laws  and  gains  implemented 
in  the  specified  flight  control  modes.  Approach  the 
close-in  closed  loop  task  in  a  conservative  manner  and 
determine  the  ability  of  the  pilot  to  refuel  utilizing 
conventional  and  decoupled  control  modes . 

3.  Detailed  procedures  are  not  specified  so  as  to 
encourage  an  unbiased  evaluation  of  the  utility  of  the 
various  DFCS  modes . 
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Handling  Qualities  Purine  Tracking  (HQDT) 


1.  Confirm  flight  control  mode,  gun  camera  operation', 
F-stop  and  film  magazine  No.  with  test  control. 

2.  Standby  Reticle  selected,  depression  angle  set  as 
directed  by  test  control  and  HUD  camera  selected. 

3.  Tracking  pilot  should  trim  for  level  flight  prior  to 
the  maneuver.  Do  not  retrim  during  the  maneuver. 
Target  pilot  should  set  up  on  speed  and  altitude  and 
stabilize  in  a  30*  bank. 

4.  Tracking  pilot  close  to  1,500  ft.  range. 

5.  Data  “ON".  In  level  flight  accomplish  a  bias 
calibration  for  each  film  magazine  by  recording  a 
short  (1  sec)  burst  of  film  with  the  pipper  on  the 
airapoint . 

6.  Tracking  pilot  call  “Start  Maneuver." 

7.  The  target  pilot  should  begin  either  a  slow  wind-up 
turn  or  a  constant-turn  at  the  test  conditions  as 
specified.  For  the  wind-up  turn,  increase  load  factor 
at  an  approximate  rate  of  1  "g"  per  10  second  to  reach 
about  5  "g's“  in  40  seconds.  Hold  5  "g's"  for  5 
seconds.  Descend  and  use  afterburner  as  necessary  to 
maintain  constant  Mach  number.  For  the  constant-turn, 
the  target  should  establish  a  constant  altitude  turn 
at  the  conditions  specified.  Use  afterburner  as 
necessary  to  maintain  level,  flight  and  constant  Mach 
number. 

8.  Tracking  pilot  squeezes  the  trigger  to  start  the 
camera  and  briskly  moves  pipper  on  to  target  from  50 
MIL  reticle  or  farther. 

9.  Tracking  pilot  must  persistently  track  the  precision 
aimpoint  agreed  to  prior  to  the  flight,  using  the 
specified  controllers.  Attempt  to  hold  1000  to  2000 
ft.  range  during  the  run. 

10.  Target  pilot  to  call  "Terminate"  45  seconds  after 
starting  maneuver. 

11.  At  the  call  "Terminate",  the  tracking  pilot  should 
acknowledge  and  briskly  move  the  pipper  at  least  50 
MILS  off  the  target  and  release  the  gun  trigger. 

12.  Data  "OFF." 

NOTE:  Either  pilot  may  terminate  the  maneuver  early  if 

required  by  unusual  or  unsafe  conditions.  ‘‘The  tracking 
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pilot  is  responsible  for  maintaining  safe  separation. 
During  the  maneuver  the  target  pilot  is  responsible  to 
maintain  clearance  from  other  aircraft. 

13.  Call  gun  camera  film  remaining  to  test  control. 

14.  Change  magazine  if  necessary  and  give  magazine  number 
to  test  control. 

NOTE:  Testing  may  indicate  that  some  of  the  conditions  in  the 

run  table  are  of  more  interest  than  others.  Therefore, 
the  run  table  should  be  viewed  as  a  "menu"  to  select 
the  specific  test  conditions.  It  is  desirable  to  have 
3  pilots  fly  2  missions  each  of  the  HQDT  maneuvers. 

Air-to-S urface  Tracking,  Strafe 

1.  Select  a  reasonable  strafe  target  (car,  plane,  strafe 
panel  etc.  )  on  the  bombing  range  and  set-up  inbound  at 
the  conditions  specified  in  the  run  table. 

2.  Perform  a  simulated  strafe  run  by  attempting  to  keep 
the  CCIP  pipper  on  the  target  during  the  run. 

3.  Use  of  decoupled  options  will  be  as  specified  in  the 
preflight  briefing. 

4.  The  pilot  shall  simulate  a  one  second  burst  when  in 
range  and  execute  a  pull-up. 

Air-to-S urface  Tracking,  Bomb 

1.  Set-up  inbound  to  the  target  at  3500'  AGL. 

2.  Upon  reaching  the  point  where  the  target  is  10*  below 
the  horizon,  pushover  and  track  the  target  with  the 
flight  path  marker. 

3.  Use  only  the  controllers  specified  in  the  run  table. 

4.  Recover  from  the  dive  at  a  safe  altitude. 

ILS 

1.  Conduct  an  ILS  approach  to  landing  utilizing  standard 
ILS  procedures. 

2.  Qualitatively  evaluate  the  aircraft  handling  qualities 
during  the  ILS . 

3 .  The  pilot  shall  use  his  normal  technique  when 
performing  this  rest.  (i.e.-  HUD  v,s  ADI,  AOA  as 
desired  etc. ) 
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Table  0.2-2 


RUN  AS/  KXT.  DFCS 

NO.  HACK  Al.T.  CONFIG.  LOAD  HOPE  PRIORITY  MANEUVER 


SC-550  400kcaa  -  Cll  A/S 

SC-551  400kcau  - 

SC-552  460kcaa  - 

SC-553  460kcaa  - 

SC-554  520kcas  - 

SC-555  520kcaa  -  CK  A/S 

SC-560  400kcaa  3.5k  CR  A/S 

SC-561  I 

SC-562  [ 

SC-563  | 

SC-564  400kcaa 
SC-565  500kcas 


SC-566 

SC-567 

SC-560 


SC- 

-569 

SOOkcau  3.5k 

CR 

A/S 

SC-570 

ll*« 

PA 

A/A 

SC-571 

13*a 

PA 

A/A 

SASQ  1  Air-to-Surface 

DASG  1  Air-to-Surfaca 

SASQ  1  Alr-to-Sur face 

DASG  1  Alr-to-Sur face 

SASG  1  'Alr-to-Sur face 

DASG  1  Air-to-Surface 

SASO  1  Air-to-Surfaca 

SASIl  Air-to-Surface 

DASH  Air-to-Surfaca 

DASU  Air-to-Surfaca 

DASH  Air-to-Surface 

SASO  Air-to-Surfaca 

SASO  Air-to-Surfaca 

DASO  Alr-to-Sur face 

DASO  Air-to-Surfaca 

DASU  1  Air-to-Surface 

SNKM  1  IDS  l  Landing 

SWUM  1  ILS  k  Landing 


Gun  Tracking 
Gun  Tracking 
G  in  Tracking 
Gun  Tracking 
Gun  Tracking 
Gun  Tracking 

Bomb  Tracking  Stick  Only 
Bomb  Tracking  Stick  &  Pedals 
Bomb  Tracking  Stick  Only 
liomb  Tracking  Stick  A  Throttle 
Bomb  Tracking  Stick  A  Pedals 
Bomb  Tracking  Stick  only 
Bomb  Tracking  Stick  A  Pedals 
Bomb  Tracking  Stick  only 
Bomb  Tracking  Stick  A  Throttle 
Bomb  Tracking  Stick  A  Pedals 


SC-575  0.9  20k  CR  A/A  SAAG  1 

SC-576  SAAG 

SC-577  SAAG 

SC-570  DAAG 

SC-579  DAAG 

SC-500  DAAG 

SC-501  0.9  DAAG 

SC-5U2  0.6  SAAG 

SC-503  I  SAAG 

SC-504  |  SAAG 

j  DAAG 

SC-506  DAAG 

SC-507  I  DAAG 
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APPENDIX  B 


8.7  WEAPON  SEPARATION  MANEUVERS 

8.7.1  Test,  Objectives 


To  meet  the  design  goal  of  improved  weapon  delivery  accuracy 
combined  with  increased  survivability,  the  AFTl/F-16  program  will 
require  that  weapons  be  released  from  the  aircraft  in  a  variety  of 
attitudes  and  accelerations  not  common  to  the  F-16.  It  is  the 
objective  of  the  weapons  separation  testing  to  provide  store 
separation  clearance  for  the  most  critical  flight/release 
conditions  anticipated  for  the  AFTl/F-16  program.  This  will  be 
accomplished  by  ejecting  BDU-33  D/B  practice  bombs  from  SUU-20 
dispensers  at  specified  conditions. 

8. 7.1.1  Test  Approach.  Data  reduction  by  the  photo-imaging 
techniques  used  in  the  F-16  FSD  program  (photo  data  analysis 
system)  will  be  required  to  provide  store  separation  clearance  for 
the  AFTI/F-16  aircraft.  Correlations  of  the  flight  test  data  and 
the  predicted  separation  characteristics  will  be  made  after  each 
flight  before  moving  to  a  more  critical  separation  condition.  The 
flight  test  conditions  will  be  adjusted  if  necessary  to  assure 
adequate  aircraft  safety.  These  steps  will  be  continued  until 
satisfactory  store  separation  has  been  verified  at  the  most 
critical  flight  and  release  condition  anticipated  for  the  AFTI/F-16 
program.  Weapon  separation  clearance  drops  will  be  made  at  higher 
lateral  accelerations  and  angles  of  sideslip  than  listed  in  Table 
8.7-1  if  Weapon  Effectiveness  objectives  indicate  a  payoff  and  if 
clearance  approval  for  the  higher  levels  is  obtained  from 
AFATL./DLJC.  The  higher  levels  to  be  considered  as  a  goal  are  +2.0g 
sideforce  and  +8.0  degrees  sideslip. 

In  most  cases  one  subcritical  release  will  be  required  before 
a  critical  release  is  executed.  In  the  case  of  canard  deflection, 
however,  the  level  of  uncertainty  is  much  higher  with  respect  to 
flow  field  changes  at  the  SUU-20.  For  that  reason,  up  to  three 
subcritical  separations  may  be  required  before  each  critical  drop. 

The  estimated  data  turnaround  time  between  subcritical  and 
critical  separation  .flights  is  now  projected  to  be  approximately 
2-1/2  to  3  weeks.  This  period  will  allow  for  film  processing 
(AFFTC  Edwards  AFB ) ,  data  extraction  (PDAS,  Pt.  Mugu)  and 
trajectory  analyses  (GD/FW).  In  some  cases  a  simple  review  of 
inflight  drop  films  will  suffice  for  clearance  of  subsequent 
flights.  In  most  cases  the  data  can  be  pipelined;  i.e.,  several 
data  sets  in  review/analysis  process  simultaneously  so  that 
separation  flights  evaluating  different  effects  can  be  inter-leaved 
or  combined  without  delaying  flight  scheduling.  However,  in 
planning  for  significantly  critical  drops,  the  full  2-1/2  to  3 
weeks  interval  should  be  allowed. 

Following  any  store  separation  build-up  test,  the  contractor 
(with  Air  Force  concurrence)  will  issue  a  clearance  to  proceed  to 
the  next  planned  test  in  a  test  series  after  data  analyses  of 
onboard  film  and/or  magnetic  tape  have  been  performed. 
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8.7.3  Test  Conditions 


The  test  conditions  for  the-  weapon  separation  clearance 
program  are  contained  in  Table  8.7-1.  Each  test  point  requires 
that  all  six  bombs  from  the  SUU-20  be  released  in  ripole  fashion. 
This  will  be  accomplished  by  selecting  the  "RIPPLE"  mode  of  SUU-20 
operation.  Runs  that  call  for  asymmetric  release  conditions,  (Bank 
angle  ^  0‘  or  180’,  Ny  £  0,  or  /?  r  0)  shall  require  that  bombs  be 
dropped  from  both  wings  simultaneously. 

8 «  7 . 2  Test  Maneuvers  and  Procedure s 

. .  seFaration  flight  testing  is  not  usually  thought  of  as  a 

discipline  requiring  the  use  of  classical  flight  test  maneuvers. 

This  is  because  data  prior  to  release  is  relatively  unimportant 
compared  to  data  at  and  immediately  after  release  which  defines  the 
aircraft  and  store  separation  characteristics. 

However,  AFTl/F-16  store  separation  tests  at  specified  Mach 
no. ,  altitude,  normal  acceleration,  dive  angle,  bank  angle,  canard 
an9l®  and  lateral  acceleration  not  only  require  orecise 
flying  but  a  high  degree  of  timing  so  that  the  release  and  impact 
occur  on  range.  (Radar  vectoring  and  space  positioning  will  be 
utilized  to  ensure  proper  range  safety) . 

Several  separation  tests  require  releases  in  a  60  degree  dive 
angle  at  0.9  Mach  and  at  the  highest,  practical  a.  Altitude  loss 
during  dive  recovery  (5500  feet)  was  obtained  by  assuming  throttle 
to  idle  after  release,  speed  brakes  fully  extended  (no  canard 
toe-in)  and  a  4  "g"  recovery  (nominal  S.L.  temperature  -  25 *C).  To 
this  value,  1,500  feet  was  added  to  account  for  ripple  release 
time,  pilot  reaction  time  and  time  to  roll  the  aircraft  to  a 
satisfactory  recovery  bank  angle.  Another  1500  feet  was  added  to 
provide  required  minimum  ground  clearance.  Thus  an  initial  weapon 
release  altitude  of  8,500  feet  AGL  is  obtained.  This  altitude  was 
also  used  as  the  release  altitude  for  related  build-up  separation 
at  30  degree  dive  angles. 

A.  Level  Flight  Release 


1.  Stabilize  at  the  flight  conditions  specified  and 
select  specified  flight  control  mode. 

2.  When  specified,  extend  speedbrakes  to  obtain  the 
proper  canard  deflection.  Use  power  as  required. 

3.  When  on  conditions  release  one  SUU-20  load  of  BLU-33s  . 
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B .  Direct  Lift  Command /Level  Flight  Release 


1.  Stabilize  at- -a  preselected  Mach  and  altitude  above 
release  altitude.  Select  specified  flight  control 
mode. 

2.  Conduct  a  constant  Mach  (approximately  -20*)  dive. 
Pull  out  from  the  dive  with  full  nose  up  direct  lift 
command  such  that  the  aircraft  is  level  at 
approximately  10k. 

3.  Maintain  full  nose  up  direct  lift  command  and  command 
bomb  release  as  the  nose  passes  through  the  horizon. 

C.  Release  in  a  Stabilized  Dive 


1.  Select  the  specified  flight  control  mode  and  extend 
the  speedbrakes  to  the  specified  canard  toe-in  angle 
(if  specified)  at  a  preselected  altitude. 

2.  Roll  into  a  dive  at  the  preselected  altitude  and  Mach 
number. 

3.  Quickly  establish  dive  angle  and  make  power 
adjustments  to  control  Mach  number  such  that  release 
may  be  commanded  at  specified  dive  angle  Mach  number 
and  altitude. 

D .  Non-Wings  Level  Release 

1.  Stabilize  briefly  at  the  specified  flight  conditions, 
select  specified  flight  control  mode  and  extend  speed 
brake  to  full  toe-in  angle.  Use  power  as  required. 

2.  Pull  up  into  a  climb,  roll  to  the  specified  bank 
angle,  pull  to  +1.5  "g"  (Nz)  and  allow  the  nose  to 
drift  down  through  the  horizon. 

3.  When  bank  angle  and  Nz  are  on  conditions  and  pitch 
angle  is  between  0*  and  -30*,  command  release. 

,4.  Maintain  flight  conditions  for  3  seconds,  roll  and 
increase  "g"  to  vector  aircraft  away  from  bomb 
trajectory. 

E .  Level  Flicht/Flat  Turn  Release 


1.  Stabilize  at  the  specified  flight  conditions  and 
select  specified  flight  control  mbde. 

2.  Apply  left  rudder  pedal  slowly  and  stabilize  at  the 
specified  lateral  acceleration  (Ay).  Maintain  wings 
level . 

3.  When  on  conditions,  command  the  release. 
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P.  Stabilized  Dive/Flat  Turn  Release 

1.  Select  the  specified  flight  control  mode  and  briefly 
stabilize  at  a  preselected  altitude. 

2 •  Roll  into  a  dive  at  the  preselected  altitude  and  Mach 
number. 

3.  Quickly  establish  dive  angle  and  make  power 
adjustments  to  control  Mach  number.  Apply  left  rudder 
pedal  and  stabilize  at  the  specified  lateral 
acceleration  (Ay). 

4.  Command  release  at  specified  dive  angle,  Mach  number. 
Ay  and  altitude . 

G.  Non-Wings  Level /Flat  Turn  Release 

1.  Stabilize  briefly  at  the  specified  flight  conditions 
and  select  specified  flight  control  mode. 

2.  Pull  up  into  a  climb,  roll  left  to  the  specified  bank 
angle,  pull  to  1.5  "g"  (Nz),  apply  left  rudder  pedal 
and  stabilize  at  the  specified  lateral  acceleration 
(Ay) . 

3.  Allow  the  nose  to  drift  down  through  the  horizon. 

4.  When  bank  angle,  Nz  and  Ay  are  on  conditions  and  pitch 
angle  is  between  0*  and  -30*,  command  release. 

5.  Maintain  flight  conditions  for  3  sec.,  release  rudder 
pedal,  roll  and  increase  "g"  to  vector  aircraft  away 
from  bomb  trajectory. 

H.  Direct  Lift  Command/Flat  Turn/Level  Flicht  Release 

. . .  *"f  '  ■■■■■■  . .  -..■■■  - - ■■■■  *  ■  ■ 

1.  Stabilize  briefly  at  a  preselected  Mach  and  altitude 
above  release  altitude.  Select  the  specified  flight 
control  mode. 

2.  Conduct  a  constant  Mach  (approximately  -20*)  dive. 

Pull  out  of  the  dive  with  full  nose  up  direct  lift 
command  such  that  the  aircraft  is  level  at 
approximately  10k.  Prior  to  attaining  level  flight 
apply  left  pedal  command  to  attain  specified  lateral 
acceleration  (Ay) . 

3.  As  the  nose  passes  through  the  horizon  with  full  nose 
up  direct  lift  command  and  at  specified  Ay,  command 
bomb  release. 
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Stabilized  Dive/Direct  LI  ft  /Flat.  Turn  Release 

1.  Stabilize  briefly  at  a  preselected  Mach  and  altitude 
and  select  specified  flight  control  mode. 

2 .  Conduct  a  60*  dive  such  that  the  specified  flight 
conditions  and  flight  control,  system  inputs  occur  at 
approximately  8. 5k  AGL. 

3.  Prior  to  reaching  8.5k  command  full  nose-up  direct 
lift  command  and  left  pedal  command  to  0.5  "g"  Ay. 

4.  At  approximately  8.5k  AGL  with  full  nose-up  direct 
lift  command  and  0.5  "g"  Ay,  command  bomb  release. 

5.  Release  L/H  controller  and  pedal  inputs  and  recover 
from  dive. 


8. 7.2.1  Maneuver  Tolerances.  Selection  of  parameter  limits 
in  flight  test  is  always  difficult  since  too  stringent  tolerances 
can  make  the  maneuver  almost  impossible  for- the  pilot  to  fly  and 
too  wide  a  tolerance  can  result  in  the  collection  of  invalid  data. 
In  either  event  a  costly  repeat  of  a  run  can  result... 


The  tolerances  set  forth  below  are  proposed  as  a  coal  so  as  to 
encourage  the  performance  of  precise  tests  with  resulting  valid 
data.  Only  the  magnitude  of  each  tolerance  is  shown ;  the  sign 
convention  does  nor.  refer  to  direction. 


Bank  angle 
Normal  Load  Factor 
Lateral  Load  Factor 
Sideslip  Angle 
Canard  Toe-in  Angle 
Roll  Rate 
Pitch  Angle 
Mach  Number 
Altitude 


+0.2” a" 
+0.2  ,'”-0g 
+1* 

+1  * 


O'/sec 
+5  * 

+y  03 

+1.000  feet 


(+15°  non  wines  level) 
(+0.5  at.  "g’s"  >  2.0) 


8. 7.2.2  Maneuver  Limits.  The  following  maneuver  limits  are 
set  up  as  a  guide  for  clearing  the  AFTI/F-16  for  BDU-33  weapon 
releases.  Upon  completion  of  Table  8.7-1,  the  AFTI/F-16  will  be 
cleared  to  release  BDU~33s  within  the  following  limits: 


Wings  Level  Non -Wings  Level 


Bank  angle 
Normal  Load  Factor 
Lateral  Acceleration 
Angle  of  Sideslip 
Canard  Toe-in  angle 
Roll  Rate 
Angle  of  A.ttack 


0* 

0,5  -  5.0  “g 
+0 . 5 "g" 

+1.0* 

0-2  7  0 
0  * /sec 
+10* 


+180* 

T.5  -  5 .  C'g 
+0 . 5  "!g  " 
+1.0° 

0-27* 

0®/sec 

+10° 
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Table  B.7-1  WEAl*ON  RELEASE  CONDITIONS 
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APPENDIX  C 


RESPONSE  TO  TURBULENCE  MANEUVER 


One  of  the  FVI  evaluation  maneuvers  listed  in  Table  1  is  the  response  to 
turbulence  maneuver.  This  maneuver  will  be  used  to  evaluate  the  gust 
alleviation  characteristics  of  five  of  the  flight  control  modes  at  two  air¬ 
speeds  each.  Gust  alleviation  differs  in  each  mode  in  the  pitch  axis 
only.  The  gun  modes  resist  pitch  rate  upsets,  the  bcrmbing  modes  resist 
flight  path  disturbances,  and  the  normal  modes  provide  improved  ride 
qualities  at  the  pilot  station. 


The  maneuver  is  approximately  three  minutes  long  to  collect  enough  data 
to  perform  a  spectral  analysis.  The  piloting  task  is  to  keep  the  wing 
level  (bank  angle  regulation)  While  minimizing  longitudinal  control  in¬ 
puts.  This  may  allow  an  analysis  of  pilot  dynamics  for  the  bank  angle 
regulation  task  for  ten  different  runs. 

The  vertical  gust  velocity  (disturbance)  time  history  will  be  calculated 
frcm  angle  of  attack  and  normal  acceleration  or  INS  vertical  velocity. 
The  power  spectral  density  of  vertical  gust  velocity  and  responses  of 
normal  acceleration  and  pitch  rate  will  then  be  computed  and  combined  to 
produce  frequency  response  functions.  In  the  lateral  axis  the  pilot  de¬ 
scribing  function  of  lateral  stick  force  to  bank  angle  error  may  also  be 
computed. 


The  flight  conditions  will  all  be  on  the  same  flgiht  over  the  same  terrain 
at  one  thousand  feet  above  ground  level  for  each  Mach  number.  Test  con¬ 
ditions  are  detailed  in  the  following  table. 


RUN 

NO. 

AS/ 

MACH 

ALT 

CONFIG 

EXT. 

LOAD 

DFCS 

MODE 

PRIORTY 

GR-01 

400KCAS 

1KAGL 

CR 

A/A 

SNRM 

1 

GR-02 

400KCAS 

1KAGL 

CR 

A/A 

SAAG 

1 

GR-03 

400KCAS 

1KAGL 

CR 

A/A 

SASB 

1 

GR-04 

400KCAS 

1KAGL 

CR 

A/A 

DAAG 

1 

GR-05 

400KCAS 

1KAGL 

CR 

A/A 

DASB 

1 

GR-06 

520KCAS 

1KAGL 

CR 

A/A 

SNRM 

1 

GR-07 

520KCAS 

1KAGL 

CR 

A/A 

SAAG 

1 

GR-08 

520KCAS 

1KAGL 

CR 

A/A 

SASB 

1 

GR-09 

520KCAS 

1KGAL 

CR 

A/A 

DAAG 

1 

GR-10 

520KCAS 

1KGAL 

CR 

A/A 

DASB 

1 

730 


C-l 


APPENDIX  D 


9.  D  F  C  S  EVALUATION  PHASE 


The  DFCS  evaluation  test  objectives  are  to  (1)  conduct  an 
evaluation  of  the  operational  usefulness  of  manual  control  of  DFCS 
task  tailored  inodes  and  (2)  conduct  an  evaluation  of  the  capability 
of  the  aircraft  to  meet  the  air-to-air  and  air-to-surface  mission 
performance  requirements.  These  objectives  will  be  accomplished  on 
dedicated  evaluation  flights  and/or  as  secondary  objectives  on  DFCS 
development  flights  (i.e.,  remaining  portions  of  weapon  separation 
flights) .  Priority  will  be  given  to  A/S  bombing. 

The  basic  approach  to  evaluation  testing  will  be  as  defined  in 
this  section.  The  scheduling  (when  to  test)  and  test  scope  (what 
to  test)  will  be  determined  by  the  AFTI/F-16  Joint  Test  Force 
(JTF),  with  AFTI/F-16  ADPO  concurrence,  based  on  the  following 
factors:  (1)  level  of  maturity  of  the  specific  DFCS  modes,  (2) 
results  of  DFCS  development  tests,  (3)  results  of  pilot  experience 
during  DFCS  development  tests  and  (4)  flight  time  available.  In 
general  DFCS  evaluation  tests  will  have  a  lower  priority  than  DFCS 
development  tests  required  to  clear  the  flight  envelope  for  the 
AMAS' flight  test  phase. 
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9.1  DFCS  EVALUATION  AND  OPERATIONS  DEVELOPMENT 


9.1.1  Test.  Obj active s  /Approach 

The  objective  of  the  tests  contained  in  this  section  is  to 
determine  the  operational  usefulness  and  evaluate  the  mission 
capabilities  of  the  various  AFTI/F-16  technologies .  This  will  be 
accomplished  by  flying  simulated/actual  air-to-air  and  air-to- 
surface  weapon  delivery  runs,  emphasizing  use  of  the  AFTI  -  unique 
flight  control  modes  and  PVI  advancements .  In  that  the  Integrated 
Fife  and.  Flight  Control.  (T.FFC)  system  will  not  be  installed  until 
the  A.MA5  phase  of  testing,  only  manually  flown  profiles  are 
included . 

The  emphasis  of  the  DFCS  Evaluation  and  Operations  Development 
tests  is  on  "pipper  effectiveness";  that  is,  the  relative  degree  of 
ease  or  difficulty  in  using  the  aircraft  controls  to  position  the 
aircraft,  and  in  turn,  the  pipper.  Runs  are  defined  using  both 
conventional  maneuvering  techniques  and  decoupled  control  mode 
options.  Successful  accomplishment  of  the  tests  outlined  herein 
should  yield  pilot  recommendations  with  supporting  quantitative 
data  on  relative  worth  of  the  various  DFCS  modes  and  PVI. 
improvements  as  they  relate  to  manual  weapon  deliveries . 

9.1.2  Test  Maneuvers  and  Procedures 


The  runs/maneuvers  to  be  used  during  the  DFCS 
Evaluation/operations  Development  'tests  are  divided  into  three 
areas:  (1)  air- to- surface  bombing,  (2)  air-to- surface  gunnery,  and 

(3)  air-to-air  gunnery.  Not  all  of  the  runs  call  for  use  of 
decoupled  modes  to  maneuver  the  aircraft  to  the  point  of  weapons 
release.  Some  runs  are  designed  to  evaluate  potential  improvements 
using  only  conventional  maneuver  techniques  that  may  be  realized 
with  the  various'  DFCS.  modes.  All  runs  described  herein  require  a 
fully  operational  avionics  system.  Details  on  the  switchology  and 
avionics  operating  procedures  required  to  use  each  weapon  delivery 
mode  will  be  included  in  the  daily  run  cards.  Additional 
information  on  these  modes  can  be  found  in  the  F-16  Avionic  System 
Manual  (Block  10). 

Of  least  importance  in  the  accomplishment  of  the  runs  in  this 
section  is  the  need  to  precisely  meet  the  dive  angles,  airspeeds, 
altitudes,  etc.,  specified  in  the  tables.  These  numbers  are 
included  only  so  as  to  define  a  range  for  parameters  that  may  have 
a  bearing  on  the  effectiveness  of  a  given  mode  to  perform  a 
specific  task.  In  cases  where  it  becomes  apparent  that  varying 
airspeed,  dive  angle  etc.,  will  have  no  effect  on  the  results,  the 
remaining  runs  should  be  deleted.  Conversely,  if  it  appears  that 
particular  modes  would  be  useful  at  conditions  other  than  those 
specified  in  the  run  table,  the  run  tables  will  be  revised. 


9. 1.2.1  Air-to-Surface  Bombing 


1.  Flat  Turn  Effectiveness  -  The  aircraft  shall  be  vectored 
inbound  to  the  target  along  a  ground  track  that  will  posi¬ 
tion  it  at  the  range  and  target  angle-off  specified  in  the 
run  table.  Upon  reaching  this  point,  the  pilot  should  be 
stabilized  on  conditions  (speed,  heading,  and  altitude). 
Test  control  will  call  "maneuver"  and  the  pilot  will  imme¬ 
diately  use  the  specified  technique  to  turn  the  aircraft 
and  place  the  CCIP  bomb  fall  line  (BFL)  over  the  target. 
The  pilot  should  continue  to  "track"  the  target  with  the 
BFL  and  fly  the  run  through  to  simulated/actual  weapon 
release.  The  maneuver  techniques  to  be  used  are  as 
follows : 

a.  Flat  Turn:  Full  input  flat  turn  (coupled  or 
iiecoupled )  to  align  with  the  target  with  pitch  and 
roll  stick  inputs  sufficient  to  maintain  a  wings 
level,  constant  altitude  maneuver. 

b.  Conventional :  Target  alignment  accomplished  strictly 
using  pitch  and  roll  stick  inputs . 

c.  Combined:  Combined  use  of  flat  turn  and  conventional 
techniques  in  a  fashion  determined  by  the  pilot  to 
most  rapidly  line-up  on  the  target.  (Example,  roll 
and  pull  for  gross  acquisition,  flat  turn  for  fine 
tracking  inputs.) 

The  primary  goal  of  this  series  of  runs  is  to  determine  the 
most  effective  maneuver  technique  for  target  alignment  as  a 
function  of  heading  change  required. 


Table  9.1-1 


FLAT  TURN  EFFECTIVENESS 


RUN  NO.  TARGET  BEARING  MANEUVER  TECHNIQUE 

RANGE  ANGLE  OFF 

EV001  TBD  TBD  Flat  Turn 

002 

Conventional 

003 

Combined 

004 

Flat  Turn 

005 

Conventional 

006 

Combined 

007 

Flat  Turn 

008 

Conventional 

009 

Combined 

010 

Flat  Turn 

Oil 

Conventional 

012 

Combined 

013 

Flat  Turn 

014 

Conventional 

015  1 

'  Combined 

ACCOMPLISHED  FLIGHT 
NO. 


733 


D-3 


All  runs  shall  be  performed  at  480  knots/2KAGL,  in  either  SASB  or 
DASB.  The  option  of  using  coupled  or  decoupled  flat  turn  is  left 
to  the  pilot's  discretion.  C-band  tracking  and  HUD  film  is 
required. 

2.  Pop-Up  Attack  Evaluation  -  Set  up  inbound  to  the  target 
sufficiently  offset  to  perform  a  pop-up  attack.  Ingress 
altitude  shall  be  between  200'  and  2000’  AGL.  (Based  on 
pilot  proficiency  and  specific  run  objectives.)  At  the 
point  the  target  is  30*  angle  off,  initiate  the  pop-up. 
Upon  rolling  out  on  final,  use  flat  turn  inputs  to  line-up 
the  BFL  with  the  target.  Follow  the  run  through  to 
siraulated/actual  release.  Evaluate  the  utility  of  the 
decoupled  flat  turn  mode  in  the  fine  tracking  phase  of 
this  type  of  profile.  Repeat  the  runs  in  DASB  and 
evaluate  usefulness  of  the  direct  lift  function  on  the 
left  hand  controller  during  the  pop-up. 

Table  9.1-2  POP-UP  ATTACK  EVALUATION 


RUN  NO. 

KCAS 

AGL 

ALT 

DFCS 

MODE 

FCC 

MODE 

REMARKS  ACCOMPLISHED 

FLIGHT  NO. 

EV020 

480 

A/R 

SASB 

CCIP 

021 

550 

1 

CCIP 

022 

480 

1 

DTOS 

023 

550 

SASB 

DTOS 

024 

480 

DASB 

CCIP 

Evaluate  Direct  Lift 

025 

550 

| 

CCIP 

function  on  left 

026 

480 

1 

DTOS 

hand  controller 

027 

550 

A/R 

DASB 

DTOS 

during  pop-up 

The  use  of  coupled  or  decoupled  flat  turn  is  per  plot  discretion. 

HUD  film  and  C-band  tracking  is  required. 

3.  DTOS  (Dive  Toss)  Evaluation  -  Set  up  3000  to  5000  feet 
above  the  specified  designate  altitude  at  an  ancle  off 
such  that  a  roll-in  to  the  target  will  yield  a  dive  angle 
reasonably  near  that  specified  in  the  run  table.  Roll-in 
on  the  target,  and  once  established  on  final  in  the  dive, 
use  the  decoupled  options  of  flat  turn  and  direct  lift  to 
place  the  flight  path  marker  (FPM) ,  target  designator  (TD) 
box  combination  over  the  target.  Designate  with  the 
weapon  release  switch  and  begin  a  4g  wings  level  pull. 
Afte.r  designation,  null  out  any  steering  error  using  flat 
turn.  If  desired,  repeat  the  run  in  SNRM  using  standard 
maneuver  techniques  as  a  baseline  comparison.  Also  vary 
the  use  of  snow  plow  canards  from  run  to  run  as  specified 
in  the  run  table.  Evaluate  the  suitability  of  the 
AFTl/F-16  controls  and  displays  to  this  type  of  delivery. 
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Table  9.1-3  DTOS  EVALUATION 
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4.  CCRP  (Continuously  Computed  Release  Point)  Evaluation  - 
This  run  is  designed  to  simulate  a  blind  delivery  using 
the  radar  for  target  designation.  The  pilot  shall  attempt 
to  maintain  "eyes  in  the  cockpit"  so  as  to  best  evaluate 
the  overall  layout  and  suitability  of  the  cockpit  and 
controls  to  this  type  of  delivery  technique.  Set-up 
inbound  to  the  target  at  approximately  15  to  20  miles. 

Upon  turning  final,  select  the  appropriate  steerpoint, 
delivery  mode  etc.,  and  continue  inbound  to  the  target. 
Refine  radar  cursor  placement  (target  designation)  as 
required  throughout  the  run,  and  use  flat  turn  inputs  to 
correct  steering  errors .  Evaluate  the  use  of  direct  lift 
inputs  to  maintain  altitude.  Fly  the  run  through  to 
simulated/actual  release. 

Table  9.1-4  CCRP  EVALUATION 


RUN  NO. 

KCAS 

AGL  ALT 

DFCS  MODE 

FCC  MODE 

ACCOMPLISHED 
FLIGHT  NO. 

EV050 

400 

3k 

DASB 

CCRP* 

EV051 

480 

l 

t 

1 

EV052 

550 

3k 

DASB 

CCRP 

The  use  of  coupled  or  decoupled  flat  turn  shall  be  per  pilot 
descretion.  HUD  film,  MPD  video,  and  C-Bank  tracking  is  required. 

*An  offset  aimpoint  shall  be  used  for  this  run  with  the  pilot  FCNP 
entries  made  after  turning  final . 

5.  LADD  (Low  Altitude  Drogue  Delivery)  Evaluation  -  This 

maneuver  is  included  so  as  to  evaluate  the  pilots  ability 
to  follow  the  vertical  and  directional  steering  cues  of 
the  LADD  mode  using  pitch  stick  left  hand  controller 
(direct  lift)  and  rudder  pedal  (flat  turn)  inputs.  Before 
flight,  enter  a  pull-up  range  of  10,000  feet,  and  zero 
Quantity  MK-106  weapons.  Inflight,  set-up  15  miles 
inbound  to  the  target  and  refine  cursor  placement  (target 
designation)  in  the  expand  radar  mode.  Use  flat  turn 
inputs  to  null  out  any  steering  errors .  Monitor  the  HUD 
symbology  for  the  pull-up  anticipation  cue.  On  command 
from  the  HUD,  perform  a  4g  pull  to  a  45®  climb  angle. 

Null  out  any  vertical  steering  errors  using  pitch  stick 
and  left  hand  controller  inputs,  but  use  only  flat  turn 
inputs  to  correct  directional  errors .  Fly  the  run  through 
to  simulated/actual  release.  Evaluate  the  usefulness  of 
the  AFTI/F— 16  peculiar  controls/displays  in  this  type  of 
weapon  delivery  profile. 

Table  9.1-5  LADD  EVALUATION 

RUN  NO.  KCAS  AGL  ALT  DFCS  MODE  ACCOMPLISHED 

„  FLIGHT  NO  . 


EV055  480  200' -2000’  DASB 

EV0S6  550  200' -2000'  DASB 
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The  use  of  coupled  or  decoupled  flat  turn  shall  be  per  pilot 
discretion.  HUD  film  is  required. 

9. 1.2. 2  Air-to-Surface  Gunnery 

1.  Flat  Turn  -  Fine  Tracking  -  Set  up  inbound  to  the  target 
so  as  to  meet  the  conditions  specified  in  the  run  table. 
Perform  strafing  pass  on  a  standard  20'  x  20'  target  on 
the  Edwards  gunnery  range.  Use  decoupled  flat  turn  inputs 
to  correct  any  directional  pipper  errors .  The  pilot  shall 
evaluate  the  wide  field-of-view  (FOV)  Head  Up  Display 
(HUD)  and  the  utility  of  decoupled  flat  turn  to  make 
steering/pipper  corrections . 

Table  9.1-6  FLAT  TURN  FINE  TRACKING 


RUN  NO. 

KCAS 

DIVE 

ANGLE 

DFCS 

MODE 

FCC 

MODE 

ACCOMPLISHED 
FLIGHT  NO. 

EV060 

400 

10* 

SASG 

STRF 

061 

460 

| 

062 

520 

1 

063 

400 

15* 

064 

460 

I 

065 

520 

1 

066 

400 

20* 

067 

460 

1 

068 

520 

20* 

SASG 

STRF 

The  use 

of 

coupled  or 

decoupled  flat 

turn 

shall  be  per 

pilot 

discretion. 

HUD  film 

is  required . 

2. 

Strafe  Pitch/Ya,w  Pointing  - 

Set  up  inbound  to 

the  target 

at  the  conditions  in  the  run  table,  with  no  pitch  or  yaw 
pointing  inputs'.  The  pilot  should  intentionally  place  the 
aircraft  velocity  vector  somewhat  off  the  intended  target. 
While  prodeding  inbound,  and  especially  once  in-range, 
pitch  and  'yaw  pointing  commands  should  be  used  to  place 
the  pipper  on  the  target.  Evaluate  the  feasibility  of 
using  pitch  and  yaw  pointing  inputs  as  a  control  device 
for  strafe  pipper  positioning. 
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Table  9.1-7  STRAFE  PITCH/YAW  POINTING 


RUN  NO.  KCAS  DIVE  ANGLE  DFCS  MODE  FCC  MODE  ACCOMPLISHED 

FLIGHT  NO. 


EV070 

400 

10* 

DASG 

STRF 

071 

460 

1 

072 

520 

10* 

073 

400 

15* 

074 

460 

| 

075 

520 

15* 

076 

400 

20* 

077 

460 

1 

078 

520 

20* 

DASG 

STRF 

HUD  film 

is  required. 

3.  Flat  Turn  Effectiveness  -  Set  up  inbound  to  the  left-most 
strafe  panel  at  the  specified  airspeed;  dive  angle  as 
desired.  When  the  in-range  cue  appears  on  the  pipper, 
simulate  a  1/2  second  burst,  and  immediately  transfer 
attack  to  the  second  target  in  line  using  decoupled  flat 
turn  inputs.  If  the  above  run  is  successful,  repeat  the 
run  again  except  after  the  first  simulated/actual  burst, 
transfer  the  attack  to  the  third  target  in  line.  Repeat 
the  procedure  so  as  to  determine  the  maximum  separation 
between  primary  and  secondary  targets  for  which  it  would 
be  feasible  to  use  flat  turn  to  .make  a  second  attack  on 
the  same  pass.  If  results  are  good  at  400  knots,  repeat 
the  procedure  at  460  and  so  on  at  520  knots  to  determine 
the  effects  of  increased  speed  on  secondary  attack 
capability. 

Table  9.1-8  FLAT  TURN  EFFECTIVENESS 


RUN  NO. 

KCAS 

DIVE  ANGLE 

DFCS  MODE 

FCC  MODE 

ACCOMPLISHED 
FLIGHT  NO. 

EV080 

400 

A/R 

SASG 

STRF 

081 

460 

1 

| 

| 

082 

520 

A/R 

SASG 

STRF 

The  use  of  coupled  or  decoupled  flat  turn  shall  b6  per  pilot 
discretion.  HUD  film  is  required. 
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.  Strafe  Pitch  Pointing  -  Set  up  inbound  to  the  target  at 
the  conditions  specified  in  the  run  table.  Prior  to 
arriving  within  gun  range  of  the  target,  command,  and  hold 
in  full  nose  up  pitch  pointing.  The  emphasis  of  this  run 
is  on  pitch  pointing  authority,  rates,  and  gains? 
therefore,  the  run  may  be  made  by  flying  perpendicular  to 
a  straight  narrow  road  and  keeping  the  pipper  on  that  road 
during  closing  by  using  only  pitch  pointing  inputs.  Since 
the  aircraft  should  automatically  trim  to  the  initial 
dive  angle,  pitch  stick  inputs  should  be  minimized  to 
purify  the  pitch  pointing  evaluation.  The  pilot  should 
evaluate  the  ability  to  null  out  elevation  errors  using 
left  hand  controller  pitch  pointing  inputs.  Post  flight 
analysis  will  be  used  to  determine  more  subtle  benefits 
that  may  be  realized  in  the  strafing  task  (i.e.,  increased 
range  from  target  at  first  firing  opportunity  due  to  nose 
up  pointing,  or  increased  altitude  clearance  from  terrain 
due  to  nose  down  pointing,  etc.). 


Table  9.1-9  STRAFE  PITCH  POINTING 

RUN  NO.  KCAS  DIVE  AGL  DFCS  FCC 

ANGLE  ALT  MODE  MODE 


EV090 

400 

3 

.0 

091 

460 

1 

092 

520 

1 

093 

400 

15 

094 

460 

095 

520 

096 

400 

20 

097 

460 

1 

098 

520 

1 

099 

400 

C 

• 

100 

460 

101 

520 

102 

400 

103 

460 

104 

520 

105 

400 

106 

460 

107 

520 

0 

• 

N/A  DASG  STRF 


100* 

I 

300’ 

I 

500' 

I 

500*  DASG  STRF 


HUD  film  and  C-band  tracking  is  required. 


9. 1.2. 3  Air-to-Air  Gunnery.  Runs  defined  in  this  section  are 
in  a  slightly  different  format  than  those  of  previous  sections. 
Rather  than  specify  each  run  on  a  point  by  point  basis,  the  target 
flight  conditions  and  geometry,  mode  selection,  decoupled  option 
use,  and  tracking  tasks  will  be  selected  by  the  JTF  at  the  time  of 
the  test.  As  in  all  of  the  evaluation  testing,  runs/maneuvers 
other  than  these  currently  suggested  in  this  section  will  be 
permitted.  It  shall  be  the  JTF 1 s  responsibility  to  verify  that  the 
modified  runs  are  in  keeping  with  the  technical  and  safety 
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objectives  contained  herein.  If  necessary,  rules  of  engagement 
will  be  briefed  before  flight  to  allow  for  a  less  structured 
air-to-air  gunnery  evaluation. 


9. 1.2. 3.1  Target  Maneuvers .  The  Set  up  conditions  for 
all  air-to-air  gunnery  runs  shall  be  the  same.  The  target  shall 
initially  be  straight  and  level  at  the  specified  airspeed.  The 
AFTI/F-16  (fighter)  will  initially  be  1NM  in  trail  co-speed.  The 
fighter  shall  then  accelerate  to  a  SO  knot  closure  rate,  and  at 
3500'  range,  call  for  the  target  to  begin  his  maneuver. 


1.  Constant  g  Turn  -  On  command,  the  target  shall  begin  a 

constant  speed/g/altitude  turn  at  the  conditions  specified 
in  the  run  table.  Hold  the  turn  for  45  seconds,  call 
"terminate"  and  roll  out. 


2.  Wind  Up  Turn  -  On  command,  the  target  shall  begin'  a 

constant  speed/altitude  WUT  at  a  build-up  rate  of  lg  per  5 
seconds.  The  build-up  should  be  continued  until  the 
target  reaches  its  manpower  sustained  turn  capability  at 
the  specified  speed.  The  target  shall  continue  the  turn 
at  this  load  factor  for  another  5  seconds,  call 
"terminate",  and  roll  out. 


Turn  Reversals  -  On  command  the  target  shall  begin  a 
constant  speed/g/altitude  turn  at  'the  condition  specified 
in  the  run  table.  The'  turn  snail  be  held  until  the 
fighter  calls  "reverse".  The  target  shall  briskly  perform 
an  unloaded  roll  and  reverse  the  direction  of  turn.  The 
fighter  shall  command  three  reversals  for  a  total  of  four 
acquisition/ tracking  opportunities . 


4.  Cine  Track  -  On  command,  the  target  shall  perform  a  2g 
wings  level  pullup.  At  20*  pitch  attitude,  begin  a  roll 
to  the  left  while  maintaining  2g*s  so  as  to  arrive  at  90* 
of  bank  with  45*  of  pitch  attitude.  (Power  should  be 
increased  so  as  to  maintain  speed  and  load  factor 
throughout  the  high  pitch  attitude  portions  of  the 
maneuver.)  Continue  the  left  roll  to  135*  angle  of  bank. 
When  the  pitch  attitude  reaches  45*  nose  low,  begin  a 
roll  to  the  right  so  as  to  arrive  at  90*  of  bank  when  the 
nose  is  60*  below  the  horizon.  (The  load  factor  should  be 
increased  to  as  much  as  4g‘s  to  keep  spjsed  constant.) 
Continue  the  right  roll  so  as  to  reach  a  wings  level  state 
when  the  nose  is  on  the  horizon.  From  this  point, 
smoothly  transition  to  a  low  roll  rate,  2.5  to  3.0g  barrel 
roll.  The  maneuver  is  complete  when  the  wings  are  level. 


9. 1.2. 3. 2  Fighter  Tracking  Task.  Tracking  tasks  are 
keyed  to  evaluation  of  either  gross  acquisition  or  fine  tracking 
capabilities.  Any  of  the  three  target  maneuvers  described  above 
are  suitable  for  use  with  tracking  techniques  defined  below.  In 
all  cases,  the  pilot  should  attempt  to  place  the  pipp.gr  within  that 
area  of  the  targets  fuselage  between  the  left  and  right  wing  roots . 
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While  tracking,  range  to  target  should  be  between  1500  and  3000 
feet. 

a.  Fine  Tracking  -  Immediately  upon  the  onset  of  the  targets 
maneuver,  the  pilot  should  fly  so  as  to  maintain  the 
pipper  on  the  target  within  the  specified  range. 

b.  Gross  Acquisition  -  After  calling  for  the  target  to 
maneuver,  the  pilot  shall  delay  his  turn  towards  the 
target.  This  delay  should  be  of  duration  sufficient  to 
force  the  fighter  to  perform  a  high  pitch  rate  maneuver  to 
acquire  the  target.  The  pilot  should  then  transition  to 
fine  tracking.  After  establishing. a  steady  state  track  on 
the  target,  the  pilot  may,  if  desired,  intentionally  let 
the  pipper  fall  off  the  target  and  repeat  the  gross 
acquisition  task. 

9. 1.2. 3. 3  DFCS  Options .  As  in  the  previous  paragraphs, 
the  mode  selections  detailed  below  are  strictly  options  to  be 
chosen  from.  They  are  somewhat  listed  in  the  desired  order  of 
testing,  however,  it  is  not  necessary  to  adhere  to  this  guideline. 

a.  SAAG  Conventional  -  Use  only  pitch  and  roll  stick  inputs 
in  the  tracking  task.  (No  rudder  pedal  inputs). 

b.  SAAG  Flat  Turn  -  Use  flat  turn  inputs  as  much  as  possible 
tc  null  out  azimuth  pipper  error's,  and  pitch  stick  inputs 
for  elevation  errors . 

c.  DAAG  Conventional  -  Same  as  SAAG  Conventional. 

d.  DAAG  Pointing  -Use  pointing  inputs  to  null  out  both 
elevation  and  azimuth  pipper  errors  once  established  in 
the  fine  tracking  phase. 

e.  SNRM  Conventional  -  Same  as  SAAG  Conventional. 


APPENDIX  E 


4.  DATA  ACQUISITION  SYSTEM 


The  Instrumentation  Systems  provided  ,on  the  AFTI/F-16  airplane 
includes  the  Airborne  Data  Acquisition  System,  Airborne  Video 
System  and  the  Camera  Systems.  The  equipment  associated  with  these 
systems  is  located  in  three  (3)  areas  of  the  airplane  as  shown  in 
Figure  4-1. 


4.1  THE  AIRBORNE  DATA  ACQUISITION  SYSTEM 

The  Airborne  Data  Acquisition  System  consists  of  the  following 
subsystems : 

1.  Pulse  Code  Modulation  (PCM) 

2.  Frequency  Modulation  (FM) 

3.  Time  Correlation  (IRIG  "B" ) 

4.  Telemetry  (T/M) 

5.  Avionics  Multiplexed  Data  Bus  (AMUX) 

6.  Recording. 

These  subsystems  are  shown  as  part  of  the  system  block  diagram  in 
Figure  4-2. 

4.1.1  Basic  Airplane  Instrumentation  Provisions 


The  AFTI/F— 16  airplane  has  instrumentation  provisions  that 
include  basic  harness.es,  a  data  programming  panel,  and  a  power 
distribution  panel.  The  basic  harness  is  routed  from  remote 
airplane  areas  to  the'  main  programming  panel  for  the  airborne  data 
system.  Power  is  distributed  from  a  central  circuit  breaker  panel 
which  is  serviced  by  the  airplane  buses .  The  basic  harnesses  are 
utilized  for  transferring  power,  control  functions,  and  measurement 
signals  throughout  the  aircraft.  Most  of  the  provisions  to 
accommodate  the  instrumentation  equipment  are  located  in  the  dorsal 
fairing  area,  aft  avionics  bay,  and  the  lower  right  equipment  bay. 

The  airplane  is  equipped  with  an  instrumented  test  nose  boom 
that  replaces  the  airplanes  normal  boom  and  provides  angle  of 
attack  and  angle  of  sideslip  measurement  capability. 

4.1.2  Pulse  Code  Modulation 


The  PCM  subsystem  uses  two  (2)  Teledyne  Controls  Model 
AIFTDS-4000  Remote  Multiplex /Demultiplex  Units  (RMDU) .  One  will 
have  the  capacity  to  format  and  record  up  to  200  hardwired 
measurements  c.nd  the  other  will  handle  up  to  255  inserted  AMUX  data 
words.  Output  PCM  serial  signals  will  be  transmitted  tp  a  ground 
station  in  addition  to  being  recorded  on  the  test  vehicle. 
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Figure  4-1  AFTI/F-16  Instrumentation  Equipment 
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4.1.3  Frequency  Modulation 


The  FM  subsystem  is  used  to  record  dynamic  data  where 
frequency  response  requirements  exceed  those  of  the  PCM  system. 

.The  FM  subsystem  uses  constant-bandwidth  and  wide-bandwidth  IRIG 
(Inter-Range  Instrumentation  Group)  voltage  controlled  oscillators 
and  can  record  up  to  nine  measurements. 

4;1.4  Recorder 

'The  on-board  data  acquisition  system  uses  a  14-track,  IRIG, 
standard,  1-inch  magnetic-tape  recorder  (Ampex  AR-700) .  At  a 
recording  speed  of  15  inches  per  second  the  recorder  has  a 
frequency  response  of  250KKz.  The  recorder  uses  a  12  1/2-inch  reel 
of  magnetic  tape  to  provide  1  1/2  hours  of  recording  time.  The 
recorder  stores  data  from  PCM,  FM,  AMUX  data  buses,  time  code 
generator,  and  voice  communications. 

4.1.5  Time  Correlation 


The  time  correlation  subsystem  consists  of  an  IRIG  “B"  Time 
Code  Generator  and  a  cockpit  time  display  unit.  The  time  code, 
generator  is  synchronized  with  the  ground  station  range  master 
clock  prior  to  each  flight  of  the  test  vehicle.  The  on-airplane 
time  code  generator  provides  a  common  time  base  for  the  on-board 
recorders  as  a  modulated  1  KHz  timing  signal  for  tape,  parallel 
output  signals  for  cockpit  time  display,  and  a  100-pulse-per-secbnd 
(PPS)  timing  fair  camera  film  recording.  The  ground  station  master 
clock  data  is  recorded  with  the  telemetry  transmissions  for  time 
correlation  with  on-board  recorders  and  cameras . 

4.1.6  Telemetry 

Three  (3)  telemetry  links  are  installed  on  the  AFTI/F-16 
flight  test  airplane.  Two  (2)  links  are  used  for  PCM  and  one  (1) 
link  is  use'd  to  transmit  selected  video  signals  from  the  airplane. 
The  first  link  transmits  the  PCM  data  from  hardwired  aircraft 
measurements  and  the  second  link  transmits  the  PCM  associated  with 
selected  Avionics  Data  Bus  (AMUX)  words  along  with  time  tag 
information.  These  PCM  telemetry  links  transmit  data  to  a  ground 
facility  for  use  as  (1)  the  Primary  Data  Source  and  for  (2)  "Safety 
of  Flight"  Real  Time  Monitoring.  The  third  link  will  be  used  for 
wideband  video  signal  transmission.  This  link  will  transmit  the 
selected  video  from  the  CTVS  camera  or  one  of  the  two  (2) 
multipurpose  displays  in  the  cockpit.  IRIG  "B"  Time  is  inserted 
into  the  selected  video  signal  prior  to  being  transmitted  to  the 
ground  station. 

4.1.7  Avionics  Data  Bus  (AMUX)  Interface  Systems 

The  AFTI/F-16  Airplane  Avionics  Data  Bus  (Ref.  MIL-STD-1553 ) 
will  be  monitored  with  two  types  of  interface  assemblies  as 
described  in  the  following  paragraphs. 
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4. 1.7.1  Total  Bus  Recording .  An  interface  unit  (IH0817) 
developed  and  used  throughout  the  F-16  full  scale  development  (FSD) 
program  is  used  to  record  the  total  AMUX  data  traffic.  This  unit 
accepts  the  single  MIL-STD-1553  data  stream,  inserts  bits  fotr- word 
and  bus  identification,  provides  synchronization  words,  and 
reformats  the  data  into  five  streams  for  recording  at  reduced 
bandwidth . 


4 .1.7. 2  Interface  for  Recording  Selected  Data  Words .  The 
AMUX  data  bus  interface  subsystem  (IHQ917)  consists”  of  two  (2) 
components.  One  component  will  provide  programming  capability  to 
select  messages  of  interest  from  the  data  bus,  store  these  messages 
in  a  memory,  and  sequence  the  words  of  the  message  into  pre¬ 
selected  positions  in  the  PCM  data  stream.  The  second  component 
will  provide  the  necessary  "hand  shake"  operation  to  insert  words, 
output  from  the  first  unit  into  the  PCM  data  stream.  This 
subsystem  provides  the  capability  to  merge  selected  data  words  from 
the  AMUX  data  bus  with  the  AIFTDS-4000  PCM  System. 

4. 1.7. 3  Time  Tag  Generator.  Data  on  the  MIL-STD-1553  bus 
monitored  in  the  test” program  must  be  time  correlated  within  one 
millisecond  of  the  actual  occurrence.  A  varying  time  lag  will 
exist  between  an  event  and  its  insertion  in  the  PCM  data  stream 
because  of  data  sampling  rates,  processing  and  subcommutation  of 
the  selected  words  from  the  MIL-STD-1553  data  bus.  A  special  time 
tag  generator  (IH0916),  with  the  IH0917,  provides  the  means  to 
account  for  time  from  an  event  until  the  data  words,  are  inserted 
into  the  PCM  system  data  stream. 

4. 1.7. 4  Data  Pump.  Internal  parameters  in  the  DFCS  and  SMS 
which  are  not  normally  outputted  on  the  AMUX  can  be  accessed  for  • 
instrumentation  purposes  by  placing  them  in  special  AMUX  blocks 
called  data  pump.  Any  software  accessible  parameter  can  be 
programmed  to  be  included  in  the  data  pump  block.  These  parameters 
are  then  accessible  by  the  instrumentation  system  through  the  AMUX 
interface  system  described  in  paragraph  4. 1.7.2.  The  DFCS  and  SMS 
each  receive  data  addresses  from  the  FCC  of  all  parameters  to  be 
included  on  data  pump.  Data  pump  can  be  reprogrammed  by  altering 
the  FCC  OFP.  Since  the  FCC  has  core  memory,  OFP  changes  can  be 
made  by  loading  in  a  new  tape.  Data  pump  address  changes  can  be 
made  as  as  addendum  to  the  regular  FCC  OFP  tape. 

4.2  AIRBORNE  VIDEO  SYSTEM 

The  major  components  of  the  Airborne  Video  System  (AVS)  are 
the  Airborne  Video  Tape  Recorder  ( AVTR) ,  a  modified  Cockpit 
Television  Sensor  (CTVS)  and  a  Video  Select  Panel  (VSP) .  The 
baseline  avionics  system  is  designed  to  route  all  video  signals 
(i.e.,  radar,  sensor,  weapon,  or  threat),  except  for  the  CTVS 
video,  into  the  Programmable  Display  Generator  (PDG)  for  display  on 
the  Multi-Purpose  Displays  (MPD) .  The  video  signal  outputs  from 
each  PDG  and  the  CTVS  are  routed  to  a  Video  Signal  Select  unit  as 
shown  in  Figure  4-3.  The  Video  Signal  Select  unit  is  rgmotely 
controlled  by  th-e  Video  Select  Panel.  One  three-position  switch  on 


AIRBORNE  VIDEO  TATE  RECORDER 
(AVTRl* 
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the  Video  Select  Panel  is  used  to  select  video  (i.e.,  CTVS,  Lt  MPD, 
or  RtMPD)  for  the  Airborne  Video  Tape  Recorder  (AVTR) ,  and  an 
identical  switch  on  the  Video  Select  Panel  is  used  to  select  the 
video  signal  to  be  transmitted  on  the  telemetry  downlink. 

The  Video  Select  Panel  has  a  feature  to  override  the  selected 
AVTR  video  signal  with  CTVS  video.  If  the  HUD  override  switch  is 
enabled,  the  CTVS  video  is  automatically  switched  into  the  AVTR 
record  line  when  the  gun  is  firing  (real  or  simulated) .  An  event 
marker  is  generated  by  the  CTVS  during  gunfire  and  weapons  release. 

Prior  to  recording  on  the  AVTR  and  transmission  on  the 
telemetry  downlink,  the  selected  video  signals  are  merged  with 
1R1G  "B"  time  using  onboard  Video  Time  Insertion  (VTI)  units.  The 
inserted  time  on  the  video  is  visible  on  the  upper  part  of  the 
video  picture,  reference  Figure  4-4.  The  inserted  time  has  a 
resolution  to  one  millisecond. 

4.3  INSTRUMENTED  NOSE  BOOM 

The  instrumented  test  nose  boom  on  the  AFT1/F-16  test  vehicle 
is  interchangeable  with  the  aircraft's  production  boom.  It 
consists  of  a  specially  compensated  pitot  probe  and  vane  assemblies 
for  sensing  angle  of  attack  and  sideslip.  Normal  production 
position  error  corrections  are  removed  when  the  test  boom  is 
installed. 


4.4  CAMERA  SYSTEMS 

4.4.1  Weapons  Separation  Cameras 

The  weapons  separation  camera  system  is  composed  of  eight  (8) 
remote  cameras,  crew  station  controls,  and  indicator  lights.  The 
cameras  record  time  from  the  time  code  generator  on  the  edge  of  the 
film.  The  camera  control  panel  inserts  a  delay  to  the  weapons 
release  command,  at  the-  pilot's  discretion,  to  assure  cameras  are 
up  to  speed  at  the  time  weapons  are  released.  There  are  eight 
active  camera  stations  for  any  weapons  separation  flight.  Two 
cameras  are  located  in  each  wing  tip  dummy  AIM-9  missile.  One 
camera  is  located  in  each  strake  area  looking  aft.  One  camera  is 
located  in  each  chaff/flare  dispenser  area  looking  forward.  The 
cameras  use  16rnm  film  and  operate  at  200  frames  per  second  during 
weapons  separation.  (See  section  8.7.4). 

4.4.2  Cockpit  Camera 


A  bracket  is  provided  in  the  canopy,  behind  the  pilot's  seat, 
for  the  installation  of  a  forward  or  aft  viewing  cockpit  camera. 
The  control  switch  for  the  cockpit  camera  is  located  on  the  left 
side  of  the  pilot's  panel.  The  forward  viewing,  over  the  pilot's 
left  shoulder,  camera  is  designed  for  forward  external  viewing  and 
does  not  provide  coverage  of  controls  or  instruments.  The  aft 
viewing  camera  configuration  is  designed  to  photograph  the  spin 
chute  during  deployment. 
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Figure  4-4  Video  Time  Insertion  Capability 


Figure  4-5  AFTI/F-16  Data  Flow 


4.4*3  HUD  Camera 


A  16mra  film  camera  is  available  to  replace  the  Cockpit 
Television  Sensor  for  selected  missions.  Installation  of ‘ this 
camera  calls  for  relocation  of  the  HUD  control  panel  to  a  side 
console.  The  camera  has  manual  and  automatic  operating  provisions. 
The  automatic  mode  initiates  camera  run  with  the  first  detent  of 
the  gun  trigger  depressed.  Camera  aperature  adjustment  is 
accomplished  manually. 


4.5  GROUND  TEST  EQUIPMENT 

Ground  support  equipment  consists  of  a  test  cart  and 
calibaration  fixtures.  These  test  cart  and  surface  position 
calibration  fixtures  are  used  for  pre-  and  post-flight  checkout, 
calibration,  and  maintenance  of  the  instrumentation  data 
acquisition  system  at  the  airplane. 


General  Dynamics  has  modified  an  existing  ground  test  cart 
from  the  F-16  FSD  equipment  pool  to  incorporate  checkout  equipment 
peculiar  to  the  Teledyne  Controls,  PCM  ( AIFTDS-4000 )  system  and  the 
IH0817  and  IH0917  AMUX  interface  assemblies. 


4.6  REAL  TIME  DATA  FACILITIES 

The  NASA  DFRC  data  acquisition  and  processing  -system  will  be 
used  for  display  of  real  time  processed  data  and  generation  of  an 
engineering  units  tape  for  AFFTC  use  in  post- flight  data 
processing.  A  simplified  schematic  of  the  ground-based  equipment 
and  data  flow  is  shown  in  Figure  4-5.  Data  from  the  aircraft  that 
will  be  received,  processed,  and  recorded  at  the  ground  station 
includes:  continuous  UKF  transmission  of  pilot  voice,  selected 

video  of  either  MPD  or  CTVS,  PCM  stream  of  hardwired 
instrumentation  parameters,  and  PCM  stream  of  selected  AMUX 
parameters.  During  flight,  this  data  will  be  presented  in  the  form 
of  audio  of  pilot's  voice,  engineering  unit  strip  charts,  CRT 
digital  displays,  and  TV  monitor  display  of  downlinked  video. 
Additionally,  a  chornological  hard-copy  printout  of  avionic  and 
DFCS  discrete  events  will  be  provided  on  an  immediate  post-flight 
basis.  Early  in  the  program  NASA's  Spectral  Analysis  Facility 
(SAP)  will  be  used  to  monitor  flutter  and  aeroservoelastic  envelope 
expansion  flights  and  for  post-flight  data  processing. 
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PROPOSED  MISSION  EFFECTIVENESS  SIMULATION  DATA  PLAN 


The  following  sections  describe  the  type  of  data  to  be 
collected  and  the  analysis  of  that  data. 

6.1  DATA  COLLECTION 

6.1.1  Magnetic  Tape 

The  primary  data  collection  method  for  this  test  will  be 
magnetic  tape.  Only  raw  data  will  be  collected,  with  all  analysis 
accomplished  off  line.  The  recording  format  and  variable  are 
identical  for  all  runs. 

Prior  to  each  data  run  a  set  of  Header  Data  will  be  recorded. 
This  header  data  will  identify  the  run,  and  can  be  used  for  rapid 
data  sorting  by  run  type  or  other  header  variable.  Data  to  be, 
recorded  on  the  header  is  as  follows: 

HEADER  DATA 

1.  RUN  IDENTIFIER  -  from  the  test  plan 

2.  PILOT  IDENTIFIER 

3.  SYSTEM  IDENTIFIER 

4.  TARGET  INITIAL  LOCATION 

5.  THREAT  LOCATIONS 

When  the  simulator  is  placed  in  "GO"  the  following  parameter 
will  be  recorded  each  frame: 

FRAME  BY  FRAME  PARAMETERS 

1.  TIME  SINCE  "GO" 

2.  PIPPER  AZIMUTH  (BODY  CO-ORDINATES) 

3.  PIPPER  ELEVATION  (BODY  CO-ORDINATES) 

4.  FPM  AZIMUTH  (BODY  CO-ORDINATES) 

5.  FPM  ELEVATION  (BODY  CO-ORDINATES) 

7.  TD  BOX  AZIMUTH  (BODY  CO-ORDINATES) 

8.  TD  BOX  ELEVATION  (BODY  CO-ORDINATES) 

9.  TARGET  AZIMUTH  (BODY  CO-ORDINATES) 

10.  TARGET  ELEVATION  (BODY  CO-ORDINATES) 

11.  DIVE  TOSS  STEERING  DEVIATION 

12.  TRIGGER  DISCRETE 

13.  WEAPON  RELEASE  DISCRETE 

14.  DESIGNATE  DISCRETE 

15.  LATITUDE 

16.  LONGITUDE 

17.  ALTITUDE 

18.  YAW 

19.  PITCH 

20.  ROLL 

21.  ALPHA 

22.  BETA 
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SECONDS 

MILLIRADIANS 

MILLI RADIANS 

MILLIRADIANS 

MILLIRADIANS 

MILLIRADIANS 

MILLIRADIANS 

MILLIRADIANS 

MILLIRADIANS 

MILLIRADIANS 


DEGREES 

DEGREES 

FEET 

DEGREES 

DEGREES 

DEGREES 

DEGREES 

DEGREES 
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FRAME  BY  FRAME  PARAMETERS 


23.  Nz 

24.  Ny 

25.  AIRSPEED 

26.  PEDAL  FORCE 

27.  THROTTLE  ROTATION 

28.  STICK  PITCH  FORCE 

29.  STICK  ROLL  FORCE 

30.  THROTTLE  POSITION 

31.  TARGET  RANGE 

32.  BOMB  BUTTON  DISCRETE 

33.  FCS  MODE 

34.  SMS  MODE 

35.  DELIVERY  MODE  (FCC) 

36.  MISSION  PHASE 

37.  TARGET  LATITUDE  (A-A) 

38.  TARGET  LONGITUDE  (A-A) 

39.  TARGET  ALTITUDE  (A-A) 

40.  TARGET  YAW  (A-A) 

41.  TARGET  PITCH  (A-A) 

42.  TARGET  ROLL  (A-A) 

6.1.2  "  Merit*1  Output 

Hard  copy  print  out  of  the  "merit"  routine  will  be  available 
after  each  data  run.  This  output  will  serve  as  a  de-briefing  tool 
only.  All  later  analysis  will  be  conducted  using  raw  data  from  the 
magnetic  tape.  The  following  data  will  be  available  from  "Merit." 

1.  TOTAL  PIPPER  ERROR  at  release 
(time  average  for  gunnery) 

2.  AZIMUTH  PIPPER  ERROR  at  release 
(time  average'  for  gunnery) 

3.  ELEVATION  PIPPER  ERROR  at  release 
(time  average  for  gunnery) 

4.  BURST  LENGTH 

5.  RANGE  at  open  fire 

6.  TIME  from  "GO"  to  X  milliradians 
total  pipper  error 

7.  TIME  from  "GO"  to  Y  milliradians 
total  pipper  error 

8.  TIME  the  pipper  is  within  3,  5,  10,  25,  50,  and  100  milli¬ 
radians  (accumulation  can  begin  at  "GO",  at  a  constant 
interval  after  "GO",  or  trigger  off  of  7  above). 


KTAS 

LBS 


DEGREES 

LBS 

LBS 

% 

FEET 
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6.1.3  Strip  Charts 


Strip  charts  will  be  used  to  monitor  the  quality  of  the 
testing  and  to  trouble  shoot  any  problem  with  the  simulation.  They 
can  be  made  available  in  de-briefing  if  required.  Since  all  data 
on  the  strip  charts  will  also  be  recorded  on  magnetic  tape  we  do 
not  plan  to  maintain  a  file  of  strip  chart  data.  A  list  of  strip 
chart  variable  to  be  recorded  is  as  follows: 

[tbd3 


6.1.4  Pilot  Questionnaires 


Pilots  will  be  asked  to  fill  out  questionnaires  on.  selected 
test  runs  to  collect  qualitative  data  and  pilot  inputs  concerning 
the  cockpit  controllers,  cockpit  displays,  general  impression  of 
work  load,  and  flying  qualities. 

The  following  questionnaires  will  be  used: 

[TBD] 


6.2  DATA  ANALYSIS 

The  magnetic  tapes  of  raw  data  will  be  used  as  inputs  to 
various  f igure-of-merit  xerograms  and  the  results  will  be 
statistically  analyzed  with  respect  to  sample  variance,  mean,  and 
statistical  significance.  Results  will  generally  be  presented  in 
the  form  of  histograms.  Conclusions  from  pilot  questionnaires  will 
be  summarized,  and  where  application  questions  will  also  be 
summarized  in  the  form  of  histograms.  The  following  sections 
describe  the  f igures-of-merit  planned  for  each  experiment. 

6.2.1  Air-to-Surface  Bombing 

6. 2. 1.1  CCIP  Alignment  Time  Experiment.  The  following 
f igures-of-merit  will  be  utilized  for  this  experiment: 

1.  Time  required  to  stabilize  the  Bomb  Full  Line  (BFL)  within 
+  2  rnilliradians  of  the  desired  target. 

2.  Time  from  BFL  stabilization  until  weapon  release. 

3.  Pipper  errors  at  weapon  release  in  rnilliradians  (total, 
azimuth,  and  elevation) . 

6. 2. 1.2  CCIP  Accuracy  Experiment.  The  following  figures-of- 
merit  will  be  utilized  for  this  experiment: 

1.  Time  from  bank  angle  less  than  10°  until  weapon  release. 
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2.  Pipper  errors  at  weapon  release  in  milliradians  (total, 
azimuth,  and  elevation) . 

3.  Flight  conditions  at  release  (dive  angle,  bank  angle, 
airspeed,  altitude,  and  slant  range) . 

4.  Bomb  miss  distance  (selected  runs) 

5.  Probability  of  Survival  (selected  runs) 

6.2.1. 3  Dive  Toss  Accuracy  Experiment.  The  following 
f igures-of-merit  will  be  utilized  for  this  experiment: 

1.  Time  from  bank  angle  less  than  10°  until  "designate". 

2.  Target  Designator  (TD)  box  errors  at  "designate"  (total, 
azimuth,  and  elevation). 

3.  Azimuth  steering  errors  at  weapon  release  in  milliradians. 

4.  Flight  conditions  at  "designate"  (dive  angle,  bank  angle, 
airspeed,  altitude,  and  slant  range) . 

5.  Flight  conditions  at  weapon  release  (dive  angle,  and  slant 
range) . 

6.  Bomb  miss  distance  (selected  runs) 

7.  Probability  of  Survival  (selected  runs) 

6. 2. 1.4  Turning  CCIP  Weapon  Release.  The  following  f igures- 
of-merit  will  be  utilized  for  this  experiment: 

1.  Pipper  errors  at  release  (total,  azimuth,  and  elevation). 

2.  Flight  conditions  at  release  (dive  angle,  bank  angle, 
airspeed,  altitude,  and  slant  range). 

3.  Bomb  miss  distance  (selected  runs) . 

4.  Probability  of  survival  (selected  runs) . 

6. 2. 1.5  Operational  System  Evaluation.  The  following 
figures-of-merit  will  be  utilized  for  this  experiment: 

1.  Pipper  errors  at  release  (total,  azimuth,  and  elevation). 

2.  Flight  conditions  at  release  (dive  angle,  bank  angle, 
airspeed,  altitude,  slant  range) . 

6.2.2  Air-to-Surface  Strafe 

6. 2. 2.1  CCIP  Strafe  Accuracy  Experiment.  The  following 
figures-of-merit  will  be  utilized  for  this  experiment: 
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.  Time  from  bank  angle  less  than  10°  to  "open  fire". 

2.  Flight  conditions  at  "open  fire"  (dive  angle,  bank  angle, 
airspeed,  altitude,  and  slant  range). 

3.  Average  or  RMS  pipper  error  during  firing  (azimuth, 
elevation,  and  total) . 

4.  Average  or  RMS  bullet  miss  distance  (azimuth,  elevation, 
and  total-selected  runs) . 

5.  Expected  number  of  hits  (selected  runs). 

6.  Expected  hits  divided  by  total  rounds  fired  (selected 
runs) . 

7.  Probability  of  Survival  (selected  runs). 

8.  Percent  of  run  time  when  tracking  errors  were  less  than  3, 
5,  and  10  milliradians. 

6.2. 2.1  CCIP  Strafe  Down  Pointing  Experiment.  The  following 
f igures-of-rnerit  will  be  utilized  for  this  equipment: 

1.  Average  or  RMS  pipper  errors  (azimuth,  elevation,  and 
total) 

2.  "Open  Fire"  flight  conditions  (slant  range,  altitude, 
airspeed,  dive  angle) 

3.  "Cease  Fire"  flight  conditions 

4.  Minimum  altitude  during  pass 

5.  Burst  length 
6.2.3  Air-to-Air  Gunnery 

The  following  figures-of-merit  will  be  utilized  for  the  air- 
to-air  experiments: 

1.  Average  or  RMS  pipper  error  during  each  firing  burst 
(total,  azimuth,  and  elevation) 

2.  Burst  length 

3.  "Open  Fire"  range 

4.  Percent  of  run  time  when  tracking  errors  were  less  than  3, 
5#  and  10  milliradians. 

5.  Expected  number  of  hits  per  run  (selected  runs) 
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6.  Probability  of  kill  per  run 
(selected  runs) 

7.  Average  or  RMS  bullet  miss  distance 
(selected  runs) 

8.  Expected  number  of  hits  per  run  divided  by  total  rounds 
fired  that  run 

(selected  runs) 
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AFFTC  EXPERIENCE  WITH  PIPPER  ERROR  AS  A  HANDLING  QUALITIES  INDEX 


B.  Lyle  Schofield  and  Tom  Twisdale 
Flight  Test  Technology  Branch,  AFFTC 


During  the  workshop  we  heard  several  references  to  the  idea  of  using 
gunsight  pipper  error  -  also  called  aiming  error  -  as  a  measure  of  airplane 
handling  qualities.  The  AFFTC  now  has  nearly  eleven  years  of  extensive 
experience  with  handling  qualities  testing  using  pilot-in-the-loop  target 
tracking  maneuvers.  All  of  our  experience  indicates  that  aiming  error 
alone  is  not  a  reliable  indicator  of  handling  qualities.  In  fact  it  can 
be  very  misleading. 

Figure  1  is  a  collection  of  four  aiming  error  (pipper  motion)  histories 
with  the  associated  Cooper-Harper  rating  given  by  the  pilot.  Note  that 
the  best  tracking  was  done  with  the  airplane  that  was  assigned  the  worst 
pilot  rating.  Also  note  that  the  tracking  performance  with  the  best  pilot 
rating  does  not  differ  much  from  the  tracking  performance  with  the  worst 
pilot  rating.  Finally,  note  the  dramatic  difference  in  tracking  performance 
for  the  two  cases  with  the  same  pilot  rating.  These  two  cases  were  flown 
by  the  same  pilot,  in  the  same  airplane,  on  the  same  flight,  at  the  same 
test  conditions,  one  right  after  the  other!  These  four  time  histories 
illustrate  that  you  cannot  assess  handling  qualities  or  workload  by 
measuring  aiming  error  alone,  and  expect  consistant  results. 

Figure  2  is  a  plot  of  pilot  "workload"  versus  RMS  (root  mean  square) 
aiming  error  with  associated  pilot  ratings.  Data  are  presented  for  levels 
1,2,  and  3  lateral -directional  handling  qualities.  There  are  no 
discernible,  credible  trends  in  these  data.  One  is  not  surprised  that  a 
good  pilot  can  produce  good  tracking  results  with  an  airplane  he  rates 
poorly.  He  does  it  by  working  hard.  Neither  is  one  surprised  that  a 
good  pilot  can  produce  good  tracking  results  with  less  work  when  the  airplane 
handles  well.  There  are  data  points  in  Figure  2  which  reflect  these  kinds 
of  unsurprising  casqs.  There  are  also  some  conflicting  cases  included  in 
Figure  2.  For  example,  there  are  data  points  which  show  that  the  pilot 
tracked  as  well  with  a  level  2  or  3  airplane  as  he  did  with  the  level  1 
airplane,  but  without  expending  any  extra  "work"  to  do  so. 

We  have  put  the  terms  "work"  and  "workload"  in  quotation  marks  because 
what  we  have  measured  is  only  one  indication  of  physical  work  or  activity 
expended  by  the  pilot  during  the  task.  We  do  not  suggest  that  is  is  a 
complete  measure  of  physical  work  or  even  that  it  is  an  especially  good 
measure  of  work  or  activity.  We  do  wish  to  point  out  that  neither 
aiming  error  nor  our  measure  of  "work"  correlated  well  with  handling 
qualities  level.  Only  the  pilots'  Cooper-Harper  rating  and  qualitative 
comments  consistantly  reflected  the  actual  handling  qualities  level  of 
the  airplane. 

It  is  clear  to  us,  based  on  a  large  volume  of  experience,  that  the 
mechanisms  by  which  a  pilot  decides  whether  he  likes  an  airplane  or  not 
are  not  well  understood.  Certainly  it  would  be  a  mistake  to  rely  on  the 
tracking  performance  data  we  get  from  AFFTC  handling  qualities  testing. 
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The  key  to  evaluating  handling  qualities  still  resides  with  the  pilots' 
ratings  and  qualitative  comments.  Pilots  have  often  told  us  that,  in  their 
opinion,  the  mental  part  of  the  tracking  task  is  the  most  fatiguing, 
especially  when  the  handling  qualities  are  poor  and  considerable  compensation 
is  required.  Right  now,  the  only  way  to  account  for  this  important  part  of 
the  equation  is  through  the  pilots  subjective  evaluation. 


The  data  presented  in  Figures  1  and  2  are  not  unusual  or  worst-case 
examples.  They  are  representative  of  the  results  of  thousands  of  tracking 
runs  conducted  at  AFFTC  over  the  past  eleven  years,  using  HQDT  and  SIFT 
handling  qualities  test  and  analysis  techniques.  (HQDT  is  an  acronym  for 
Handling  Qualities  During  Tracking.  SIFT  is  an  acronym  for  System 
Identification  From  Tracking.)  HQDT, and  SIFT  were  developed  at  AFFTC  and 
are  documented  in  references  1,2,  and  3. 


We  would  be  happy  to  talk  with  you  about  our  tracking  and  handling 
qualities  experience  at  AFFTC.  We  can  be  reached  at  (805)  277-3779  or 
Autovon  350-3779. 
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AZIMUTH  ERROR  (MILS)  RZIMUTH  ERROR  (MILS) 


AZIMUTH  ERROR  (MILS) 


FIGURE  1  EXAMPLES  OF  TYPICAL  SIFT  (HQDT)  TRACKING  RESULTS 

SHOWING  HAZARDS  OF  RELYING  ON  TRACKING  ERROR  AS 
THE  ONLY  MEASURE  OF  HANDLING  QUALITIES 


759 


rudder  pedal  "work?  (or  "activity"),  (lb-sec)/sec 


Level  1  lateral-directional 
handling  qualities 

Level  2  lateral-directional 
handling  qualities 
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Level  3  lateral-directional 
handling  qualities 

Pitch  handling  qualities  are 
Level  1  for  all  cases 

Numbers  beside  symbols  indicate 
Cooper-Harper  rating 
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FIGURE  2. 


Influence  of  pilot  "workload"  (or  "activity") 
on  tracking  error  and  pilot  rating 
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SUMMARY  OF  PROCEEDINGS 


A  workshop  was  held  at  Edwards  AFB  that  brought  together 
technical  experts  in  the  disciplines  of  human  factors, 
pilot  dynamics,  pilot  modeling,  research  and  development 
and  test  and  evaluation  to  address  flight  testing  to  identify 
pilot  workload,  pilot  dynamics  and  task  performance.  The 
state  of  the  art  was  laid  out  by  the  more  than  40  presen¬ 
tations  delivered  by  representatives  of  Air  Force,  Navy, 
Army,  NASA,  FAA,  university,  private  and  foreign  organi¬ 
zations  (see  Appendix  A).  Informal  discussions  also  were 
beneficial  for  exchanging  information  and  ideas .  Based 
upon  these  inputs  and  written  comments  of  the  participants 
(Appendix  B)  the  editors  have  formulated  the  following 
broad  conclusions  and  recommendations  . 

Flight  Testing  to  Identify  Pilot  Workload . 

Workload  is  a  multidimensional  concept  in  which  the 
pilot  (physiology,  perceptions,  technique,  training),  the 
vehicle  (dynamics,  controls,  displays,  subsystems),  the 
tasks  (number,  difficulty,  relative  importance)  and  the 
environment  (stress,  disturbances)  all  play  significant 
and  interrelating  roles  .  Each  aspect  must  be  carefully 
considered  in  order  to  effectively  assess  pilot  workload 
in  flight .  One  or  several  of  these  aspects  of  pilot  workload 
have  been  objectively  measured  in  flight  by  Schiflett,  Van 
de  Graaff,  Roscoe,  the  Navy  Pacific  Missile  Test  Center 
and  the  Air  Force  School  of  Aerospace  Medicine .  Important 
measures  that  show  near-term  promise  for  assessing  pilot 
workload  in  flight  are  pilot  subjective  ratings,  rate  of 
pilot  control  activity,  heart  rate,  and  secondary  task 
performance .  Several  other  measures  were  proposed  in  the 
proceedings  .  The  most  promising  of  these  for  further 
development  appears  to  be  the  event-related  brain  potential 
described  by  Donchin  and  Biferno. 

Workload  should  be  specifically  addressed  throughout 
the  systems  acquisition  process  .  Workload  technology  prom¬ 
ises  to  become  as  useful  in  the  design,  development, 
test  and  evaluation  of  new  systems  as  flying  qualities 
technology  is  today.  Resources  should  continue  to  be 
allocated  to  measureing  pilot  workload  because  increasingly 
complex  mission  demands  continue  to  be  made  of  pilots  and 
their  aircraft . 


Flight  Testing  to  Identify  Pilot  Dynamics  and  Task 
Performance . 

These  areas  are  complementary  to  the  evaluation  of 
pilot  workload  and  thus  deserve  careful  attention.  Van  de 
Graaff  has  successfully  combined  measures  of  pilot  dynamics, 
pilot  workload,  and  task  performance  on  a  helicopter  in¬ 
flight  experiment.  Complementary  measures  for  all  these 
areas  are  listed  in  an  annotated  bibliography  of  "Pilot 
Performance"  measures  described  by  Mixon  and  Moroney. 

Models  of  pilot  dynamics  are  presently  being  used  to 
design  the  dynamic  characteristics  of  flight  controls  and 
displays.  However,  the  usefulness  of  the  models  has  been 
hampered  by  the  lack  of  validating  flight  test  data. 
Methods  for  identifying  pilot  dynamics  have  been  successfully 
used  during  simulation  but  in-flight  experience  is  lacking. 
Often  the  cues  sensed  by  the  pilot  are  difficult  to  instrument 
or  measure.  However,  it  is  still  important  to  determine 
the  strategy  and  dynamics  used  by  pilots  during  critical 
flying  tasks  to  validate  simulator  fidelity  and  aid  in 
vehicle  design. 

Task  performance  or  mission  effectiveness  measures  are 
important  in  the  design  and  evaluation  of  aircraft.  They 
are  quite  sensitive  to  variations  in  the  task  and  initial 
conditions  and  must  be  applied  and  interpreted  very  care¬ 
fully. 


RECOMMENDATIONS . 


Two  primary  technical  recommendations  for  the  research 
and  development  community  resulted  from  the  workshop. 

1.  Define  and  introduce  into  common  use  a  stan¬ 
dardized  set  of  objective  and  subjective  workload 
and  task  performance  measures  and  procedures.  Such 
a  set  will  encourage  those  conducting  experiments 
to  plan  and  report  findings  that  will  be  meaningful 
throughout  the  pilot  workload,  pilot  dynamics,  and 
task  performance  disciplines.  Not  all  the  elements 
of  the  set  would  need  to  be  measured  in  every 
experiment.  Flight  test  engineers  could  then  adapt 
the  set  to  particular  experimental  circumstances . 

2.  Hold  periodic  conferences  to  stimulate  the 
exchange  of  technical  results  among  the  disciplines 
involved  in  the  measurement  of  pilot  workload, 
pilot  dynamics  and  mission  effectiveness.  These 
conferences  will  allow  the  lessons  learned  in  the 
present  workshop  to  be  applied,  extended  and  distri¬ 
buted  widely.  Perhaps  next  year’s  Annual  Conference 
on  Manual  Control  could  be  expanded  to  meet  this 
need . 

In  addition,  flight  research  projects  should  be  under¬ 
taken  to  validate  pilot  workload  measurement  techniques 
and  models  of  pilot  dynamics.  This  is  needed  to  develop 
confidence  in  those  measurement  techniques  and  models  so 
that  they  may  be  effectively  used  by  designers  of  advanced 
aircraft . 


763 


APPENDIX  A 


ORGANIZATIONS  REPRESENTED 


Attending  the  workshop  were  152  people  from  diverse 
technical  backgrounds  and  representing  55  organizations. 
These  government,  private  and  foreign  organizations  are 
listed  here  to  allow  the  reader  to  appreciate  the  scope  and 
importance  of  pilot  workload  and  pilot  dynamics  measurement. 
Names  and  addresses  of  participants  will  be  kept  on  file  by 
the  organizers  at  the  Air  Force  Flight  Test  Center,  6520 
Test  Group/ENAH,  Stop  239,  Edwards  AFB,  CA  93523. 


GOVERNMENT 


U.S.  AIR  FORCE 

HQ  Air  Force  Test  and  Evaluation  Center,  Kirtland  AFB,  NM, 

AFWAL/ Flight  Dynamics  Laboratory,  Wright-Patterson  AFB,  OH 

Air  Force  Human  Resources  Laboratory,  Williams  AFB,  AZ  and 
Brooks  AFB,  TX 

Air  Force  Academy/DFBL,  Colorado  Springs,  CO 

Air  Force  Aeronautical  Systems  Division,  Wright-Patterson 
AFB,  OH 

4950  Test  Wing/Wright-Patterson  AFB,  OH 

Air  Force  School  of  Aerospace  Medicine,  Brooks  AFB,  TX  and 
Edwards  AFB,  CA 

Air  Force  Flight  Test  Center,  Edwards  AFB,  CA 

Air  Force  Test  Pilot  School,  Edwards  AFB,  CA 

Air  Force  Office  of  Scientific  Research,  Bolling  AFB,  D.C. 

Air  Force  Aerospace  Medical  Research  Lab,  Wright-Patterson 
AFB,  OH 

Air  Force  Systems  Command,  Andrews  AFB,  Mb 

U.S.  NAVY 

Naval  Air  Test  Center,  Patuxent  River,  MD 
Naval  Weapons  Center,  China  Lake,  CA 
Naval  Air  Systems  Command,  Washington  D.C. 

Naval  Air  Development  Center,  Warminster,  PA 
Naval  Health  Research  Center,  San  Diego,  CA 
Pacific  Missile  Test  Center,  Point  Magu,  CA 
Naval  Post  Graduate  School,  Monterey,  CA 
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GOVERNMENT 


U.S.  ARMY 

Army  Research  &  Technology  Labs  (AVRADCOM) ,  Moffett  Field, 
CA 

Army  Human  Engineering  Lab,  Aberdeen  Proving  Grounds,  MD 

NATIONAL  AERONAUTICS  AND  SPACE  ADMINISTRATION 

NASA  Ames  Research  Center,  Moffet  Field,  CA 
NASA  Johnson  Space  Center,  Houston,  TX 

NASA  Dryden  Flight  Research  Facility,  Edwards  AFB,  CA 
NASA  Langley  Research  Center,  Hampton,  VA 

F.A.A. 


Federal  Aviation  Administration,  Northwest  Region,  Seattle, 
WA 


r 


PRIVATE 


CONTRACTORS 


Lockheed-Georgia  Company,  Marietta,  GA 
Peceptronics  Inc.,  Woodland  Hills,  CA 
Rockwell  International,  Los  Angeles,  CA 
Bell  Labs,  Pisataway,  NJ 

Systems  Technology  Inc.,  Mountain  View,  CA  and  Hawthorne,  CA 

Canyon  Research  Group  Inc.,  West  Lake  Village,  CA 

Salisbury  Labs,  Worchester,  MA 

Douglas  Aircraft,  Long  Beach,  CA 

Northrop  Corp . ,  Hawthorne ,  CA 

McDonnell  Douglas  Corp.,  St  Louis,  MO 

SRI  International,  Menlo  Park,  CA 

Analytical  Mechanics  Association,  Mountain  View,  CA 

BDM  Corp.,  Dayton,  OH 

Boeing  Computer  Service,  Seattle,  WA 

General  Dynamics  Corp.,  Ft  Worth,  TX 

Alphatech  Inc.,  Burlington,  MA 

Systems  Control  Technology  Inc.,  Palo  Alto,  CA 

Systems  Research  Labs  Inc.,  Dayton,  OH 

Flight  Engineers  International  Association,  Washington  D.C. 
Bolt  Beranek  and  Newman  Inc.,  Cambridge,  MA 
Hughes  Helicopters,  Culver  City,  CA 
Honeywell  Systems  Research,  Minneapolis,  MN 
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UNIVERSITIES 


Arizona  State  University,  Tempe,  AZ 

University  of  Illinois,  Champaign,  IL 

University  of  Southern  California,  Los  Angeles,  CA 

Purdue  University,  West  Lafayette,  IN 

Wright  State  University,  Dayton,  OH 

Worcester  Polytechnic  Institute,  Worcester,  MA 

FOREIGN 

National  Aerospace  Lab  (NLR),  Amsterdam,  Netherlands 
Royal  Aeronautical  Establishment,  Bedford,  England 
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APPENDIX  B 


RESPONSES  OF  PARTICIPANTS 


On  the  final  day  of  the  workshop,  the  participants  were 
asked  to  provide  feedback  on  the  workshop  by  responding  to 
two  questions : 

1.  What  were  your  objectives  for  this  workshop?  How 
well  were  they  accomplished? 

2 .  Where  should  we  go  from  here? 

The  written  responses  have  been  edited  and  are  published 
here  to  allow  the  reader  to  appreciate  the  general  consensus 
on  the  state  of  the  art  of  flight  testing  to  identify  pilot 
workload  and  pilot  dynamics. 

QUESTION  Is 


What  were  your  objectives  for  this  workshop?  How  well 
were  they  accomplished? 

1.  "My  objective  was  to  find  out  what  the  state  of  the 
art  in  workload  measurement  was  and  what  tools  were  avail¬ 
able.  To  this  end  my  objectives  were  met." 

2.  "Would  have  liked  more  opportunity  for  discussion  - 
to  define  and  focus  attention  on  main  issues.  One  objective 
that  was  achieved  was  to  at  least  get  aircraft  and  human 
factor/psychology  types  together." 

3.  "Get  overview  of  recent  engineering  and  psychological 
approaches.  Accomplished  to  a  great  degree,  due  to  large 
sample  papers,  good  attendance." 

4.  Workshop  Objectives  of  AFTEC  Team: 

a.  Bring  attendees  up  to  the  state  of  the  art  for 
their  mutual  comprehension  and  use. 

b.  Promote  further  development  of  methods  through 
stimulation  of  thought/interaction. 

c.  Organize  and  coordinate  workload  efforts  in 
design,  testing  and  evaluation  on  a  national  basis  to 
optimize  the  return  on  all  our  efforts. 

d.  Utilize  our  voices  on  an  organized  basis  to 
promote  the  proper  addressing  of  human  factors  issues  from 
the  very  inception  of  new  systems. 
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e.  Provide  scientifically  valid/realistic  oper¬ 
ational/logistical  assessments  of  new  systems  in  terms  of 
their  operators.  ( DOD  5000.3) 

5.  "Objectives  were  to  try  to  ascertain  state  of  the 
art  in  workload  measurement,  and  significance  of  workload 
levels,  for  application  in  civil  transport  certification." 

6.  "My  objectives  in  attending  a  conference  are  usually 
twofold:  (a)  to  update  my  store  of  relevant  technical 
information,  and  to  help  others  do  the  same,  and  (b)  make 
professional  and  business  contacts.  The  technical  inter¬ 
change  was  useful,  although  not  quite  what  I  had  expected. 
Specifically,  the  format  was  more  like  that  of  a  standard 
conference  of  prepared  papers,  with  little  of  the  kind  of 
give  and  take  that  one  associates  with  a  workshop.  The 
potential  for  contacts  more  than  met  my  expectations . " 

7.  "My  objective  in  attending  this  workshop  was  to 
obtain  an  overview  of  the  technology  used  to  measure  workload 
and  model  pilots,  to  find  what  the  current  state  of  the 
art  has  to  offer  (it  really  is  an  art),  and  to  meet  those 
individuals  who  are  at  the  forefront  of  this  research. 
The  very  broad  spectrum  of  presentations  which  were  assembled 
for  the  workshop  did  indeed  meet  these  objectives.  I 
wanted  to  learn  about  those  tools  which  have  been  developed, 
tested,  and  validated  that  I  could  obtain  for  immediate  use 
on  a  Pilots  Factors  Research  contract.  Unfortunately, 
this  objective  has  not  been  totally  fulfilled.  While  I 
will  obtain  and  use  SWAT  from  AFAMRL/HE,  I  had  personally 
hoped  that  a  tested  and  validated  objective  workload  assess¬ 
ment  technique  (OWAT?)  would  be  available. " 

8.  "Objective  was  to  understand  various  facets  of  the 
workload  problem  and  to  listen  to  people  from  diverse 
backgrounds.  Both  these  objectives  were  met  very  well." 

9.  "My  objectives  for  this  workshop  were  to  receive  a 
rapid,  efficient  update  on  state  of  the  art  in  pilot 
workload.  I  use  workload  as  a  tool  in  evaluating  alternative 
design  approaches  in  applied  settings.  My  update  objective 
was  very  much  achieved.  I  was  disappointed  because  I  hoped  - 
fantasized  -  that  greater  progress  would  have  been  revealed. 
However,  I  am  nonetheless  impressed  with  the  amount  of 
attention  woYkload  is  receiving  and  am  very  hopeful  for 
the  future." 

10.  "Attendance:  to  obtain  awareness  of  current  efforts 
in  pilot  workload.  Comments:  excellent  conference  (a 
good  sound  system  would  have  been  appreciated)." 
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11.  "My  goal  was  to  find  out  about  recent  developments 
in  pilot  performance  measurement  and  pilot  workload  measure¬ 
ment.  I  am  very  satisfied  with  the  Thursday  session  which 
is  the  only  one  I  could  attend." 

12.  "Everyone  (almost)  in  Workload  and  Pilot  Models  came 
very  well  organized  (albeit  casual)  to  your  party.  Good 
instructions  and  map  (should  have  had  motels  identified 
thereon).  Good  overall  plan  and  durations:  Military  people 
with  needs  and  overview  of  work  (day  1),  Topic  A  (day  2), 
Topic  B  (day  3)  allows  specialists  to  attend  on  particular 
days.  The  name  Workshop  is  a  misnomer.  A  Workshop  includes 
several  small  groups  talking  informally  and  with  much  more 
dialogue  and  interchange.  It  is  closest  to  a  Conference 
(no  drinking)  or  Symposium  (includes  wine,  women  and  song, 
look  it  up  in  American  Heritage  Dictionary) .  More  questions 
and  interchange  would  be  desirable,  if  a  Workshop  is 
intended.  But,  for  more  coverage  in  a  conference  format 
the  idea  of  presentations  and  private  conferences  at  long 
breaks  is  a  viable  alternative." 

13.  "My  objectives  were  twofold:  to  determine  the 
orientation  of  the  approaches  being  pursued  in  comparison 
to  the  approach  I  plan  to  implement  as  part  of  our  research 
program?  to  obtain  a  feel  for  the  Air  Force  desires  concerning 
workload  assessment  as  part  of  the  design  and  flight  test 
phases  in  order  to  suggest  to  our  company  management  a 
posture  which  will  permit  us  to  more  effectively  meet  the 
requirements  of  our  primary  customer." 

14.  "I  came  to  the  workshop  to  (1)  share  my  ideas  and 
get  feedback,  (2)  get  more  ideas  on  pilot  parameter  I.D. , 
(3)  hear  about  the  AFTl/F-16  flight  test  to  see  how  we 
might  better  interact  or  support  it.  I  think  it  was  great, 
especially  the  interdisciplinary  aspects  of  it." 

QUESTION  2: 


Where  should  we  go  from  here? 

1.  "Meet  at  least  quarterly  in  support  of  the  same 
objectives.  Begin  a  standardization  of  terminology  which 
appears  to  be  in  pretty  good  shape  already.  Work  to  enhance 
the  quality  of  all  methods:  task  analysis,  computer  model¬ 
ling,  subjective  methods,  performance  methods.  Promote 
combined  methods .  Support  in  our  organizations  the  develop¬ 
ment  of  multisensor  capable  mission  area  simulator(s)  to 
replace  the  current  fragmented/piecemeal  approach  and  be 
available  to  all  developers/testers.  Support  development 
of  airborne  test  vehicles  for  advanced  manned  systems." 
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2.  "A  large  fraction  of  human  factors  related  accidents 
result  during  low  workload  levels.  I  think  the  reason  is 
related  to  low  workload,  monotony,  etc.  Hopefully,  in  the 
future  this  will  be  addressed,  because  the  problem  is 
getting  worse.  With  more  automation,  climb,  cruise,  and 
approach  phases  find  the  pilot  not  actively  in  the  control 
loop,  but  a  monitor  and/or  manager.  I'm  not  sure  he's 
very  good  at  this.  It  would  seem  that  a  major  human  factors 
problem  is  minimizing  error  rate  output.  Workload  is  only 
one  facet  of  this." 

3.  "Have  to  get  funding  -  sell  management.  Next  time, 
allow  time  for  questions.  Great  job  by  organizers  1" 

4.  "I  would  still  like  to  see  a  true  workshop  on  pilot 
workload,  but  I  don't  think  the  rate  of  technical  progress 
is  sufficiently  great  to  warrant  such  a  specialized  activity 
every  year.  Let  me  suggest  the  following  approach:  First, 
encourage  researchers  to  submit  their  latest  results  to  an 
appropriate  yearly  conference  (the  Annual  Manual  might  be  a 
good  one)  which  makes  a  point  of  always  having  a  session  on 
pilot  workload.  Second,  have  a  periodic  but  less  frequent 
(say  every  2  or  3  years)  review  of  the  field  in  which  there 
is  a  collective  attempt  to  define  the  state  of  the  field 
and  to  recommend  approaches  for  filling  knowledge  gaps. 
Such  a  review  would  divide  time  between  formal  presentations 
and  roundtable  discussion." 

5.  "Who  is  "we”?  People  interested  in  workload,  but 
different  aspects:  physiological,  objective  and  subjective 
measures,  models.  It  would  be  worthwhile  to  summarize 
their  positions/progress/status,  etc.  A  subcommittee  ap¬ 
proach  might  be  useful.  Each  subcommittee  makes  presenta¬ 
tions  to  those  of  like  interest  and  later  the  group  chairman 
of  each  group  reports  out  to  the  larger  body.  This  info 
is  exchanged  on  broader  level.  Coordinate  with  WORKLOAD 
TAG  (Technical  Advisory  Group).  Keep  operational  people 
involved . " 

6.  "About  the  conference  -  I  think  it  was  a  good-to- 
great  one  time  event.  We  already  have  Annual  Manual,  TAG 
and  SUB-TAG  groups  and  interest  groups  associated  with 
workload  at  Human  Factors  meetings.  So  there  are  already 
several  Annual  or  Semiannual  meetings  at  which  workload  is 
the  prime  or  only  topic.  A  unique  feature  of  your  meeting 
was  that  behavioral  and  control  theory  types  could  get 
together  to  compare  notes.  I  think  you'll  have  real  problems 
passing  the  meeting  on  to  others  to  manage.  The  location 
offers  an  outstanding  opportunity  to  those  poor  unfortunates 
who've  never  been  assigned  there.  I  favor  a  periodic 
meeting  like  the  one  you  organized  but  with  enough  time  in 
between  for  new  material  to  emerge,  new  needs  to  develop 
and  the  ripples  from  prior  meetings  to  die  down.  Once 
every  2  or  3  years  would  be  about  right.  Also,  with  a  bit 
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of  thinking  you  could  come  up  with  a  Thematic  variation  on 
workload  and  its  many  facets  that  could  enliven  interest 
in  the  topic.  I  think  the  flight  test  requirement  issue 
will  only  be  carried  by  folks  like  you  and  NATO  where  it 
is  a  daily  concern.  I  encourage  you  to  keep  it  a  part  of 
your  meetings." 

7.  "Every  speaker  should  provide  his/her  own  operational 
definition  of  'workload'." 

8.  "There  needs  to  be  more  pooling  of  knowledge  by  people 
from  different  DoD  and  civilian  agencies  and  more  combined 
test  efforts.  Workshops  such  as  this  should  help.  It 
seems  to  me  that  a  lot  of  people  are  independently  working 
on  similar  programs  and  that  the  field  of  human  factors 
would  benefit  from  more  consolidation  of  effort." 

9.  "Generate  needs/requirements  statement,  from  technical 
viewpoint,  required  RDT&E  WL/pilot  dynamics  funding  efforts. 
Highlight  successful  integrated  efforts  tri-service/FAA/NASA 
as  examples  for  continued  and  new  RDT&E  funding  efforts. 
Integrate  test  requirements,  e.g.,  software  into  prototype 
weapon  systems  for  flexible  changes  required  in  OT&E  and 
OPTE  for  fleet  data  acquisition  and  analysis.  Recommenda¬ 
tions  should  be  made  through  established  TAG  representatives 
which  have  the  "charter  and  blessing"  of  higher  DoD  authority 
for  implementation,  e.g.,  specs." 

10.  "Where  do  we  go  from  here?  (a)  continue  these  workshops 
on  a  annual  or  biannual  basis;  (b)  improve  audio/visual 
conditions;  (c)  increased  emphasis  on  flight  test  data  and 
flight  test  validation  of  ground  based  results.  Note; 
This  requires  increased  pressure  by  DoD  agencies  interested 
in  pilot  workload  data;  pressure  on  other  DoD  agencies  that 
influence  who  can  collect  what  kind  of  data  in  flight  test 
programs  ( RDT&fc  and'  OT&E).  Flight  Test  Data  -  non-avail¬ 
ability:  much  data,  both  qualitative  and  quantitative,  is 
being  generated  in  flight  test  operations  at  EAFB  and 
elsewhere.  These  data  are  being  developed  in  RDT&E  and 
OT&E  types  of  flight  testing.  The  vast  majority  of  Air 
Force  and  Navy  agency  reports  containing  these  data  are 
limited  distribution  to  U.S.  Government  agencies  only  that 
are  involved  in  test  and  evaluation.  The  dissemination  of 
these  data  are  very  severely  reduced  because  they  are  not 
readily  available  and  require  considerable  time  and  effort 
to  obtain.  It  should  be  noted  that  a  very  large  percentage 
of  these  limited  distribution  statements  is  not  based  on 
military  security  or  need  to  know.  Proprietary  and  other 
flimsy  allegations  are  given  as  an  excuse  for  limited 
distribution  of  data  reports.  The  cost  of  development  of 
the  flight  test  data  reports  starting  with  the  conceptual 
design  of  the  aircraft  system  years  earlier  was  paid  for  by 
the  U.S.  taxpayers.  This  group,  U.S.  taxpayers,  also  pay 
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for  the  development  of  follow-on  systems  whose  designers 
are  unable  to  benefit  from  the  mistakes  (as  well  as  good 
design  features)  made  on  earlier  systems  that  have  recently 
undergone  flight  test." 

11.  "Include  names  and  addresses  of  attendees  in  Proceedings 
to  help  people  to  keep  in  contact.  Get  proceedings  out 
quickly. " 

12.  "The  modern  cockpit  will  incorporate  many  new  control 
and  display  systems.  A  goal  in  workload  evaluation  should 
be  to  help  in  defining  what  the  modern  cockpit  should  look 
like.  We  have  the  technology  to  do  anything.  What  should 
we  do?  Basic  pilot  modelling  has  been  done  using  " S"  domain 
models.  If  one  considers  modelling  the  pilot  by  comparison 
to  a  digital  computer  (three-part)  the  model  would  be  more 
adaptable  to  the  variety  of  processes  the  human  is  capable 
of  doing. 

13.  "In  my  opinion,  eventually  we  are  going  to  have  to 
develop  absolute  workload  criteria.  Essentially,  current 
workload  measures  are  comparitive  measures,  i.e.,  which 
candidate  design  has  the  lowest  workload.  This  information 
is  useful  to  a  point,  but  the  real  question  is  not  what 
design  is  best,  instead  it  is  which  designs  have  an  'accept¬ 
able'  workload.  No  matter  how  much  better  one  design  is 
using  comparitive  measures,  there  is  no  guarantee  that  any 
of  the  designs  have  an  'acceptable"  workload.  The  deter¬ 
mination  of  absolute  workload  acceptability  criteria  will 
be  even  more  time  consuming,  difficult  and  frustrating; 
however,  I  really  think  this  is  where  we  need  to  start 
orienting  our  efforts  because  this  is  where  the  real  benefits 
of  workload  assessment  lie." 

14.  "Should  have  one  every  two  years;  1  in  west,  1  in  east 
(Pax  R?) ." 

15.  "Conferences  like  this  should  probably  be  held  every 
year.  Everyone  should  be  encouraged  to  measure  both  workload 
and  performance  simultaneously  whenever  possible." 

16.  "Suggest  consideration  of  a  steering  group  (Tri-Service 
plus  industry  representation)." 

17.  "Keep  it  going  every  one  or  two  years." 

18.  "In  terms  of  'where  do  we  go  from  here',  I  believe  that 
both  the  operational  severity  of  the  workload  problem  and 
the  breadth  of  techniques  proposed  call  for  the  following 
activities;  (a)  formulation  of  a  list  of  'action  items'  to 
address  both  short  and  long  term  problems  in  workload 
assessment/pilot  modelling.  What  needs  to  be  done?  When? 
What  is  the  most  pressing  issue?  Prioritize  all  issues  (by 
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workshop  attendee  agreement,  by  status  of  current  research, 
by  dollars  available?);  (b)  Assign  appropriate  action  items 
to  those  individuals  or  organizations  most  suited  to  perform 
them;  (c)  standardize  techniques,  methodologies,  handbooks, 
scenarios,  etc.,  and  distribute  them!  Perhaps  a  model 
repository  at  Air  Force  Flight  Dynamics  Lab  which  could 
exercise  configuration  control  would  be  useful  for  the 
efficient  accomplishment  of  a  structured  set  of  action 
items  in  an  efficient  manner;  (d)  continue  these  workshops. 
Perhaps  hold  them  in  conjunction  with  the  Annual  Manual  to 
(i)  avoid  duplication,  (ii)  increase  depth  and  breadth  of 
papers,  (iii)  save  travel  funds.  If  this  is  done,  the  pilot 
dynamics  presentations  and  the  pilot  workload  presentations 
could  be  given  over  a  4-5  day  period;  (e)  produce  a  tri¬ 
service  document  which  lists  people,  agencies,  projects, 
schedules,  and  funds.  (Similar  to  JTCG/AS  publication.) 
This  would  help  to  avoid  duplication  and  identify  gaps 
with  which  to  focus  future  research;  (f)  develop  a  state 
vector  for  pilot  workload.  This  multi-dimensional  metric 
could  be  standardized  for  use  throughout  DoD/FAA/NASA  and 
might  look  like : 


I  T  Physiological  Measures 
!  _[(i.e.,  heart  rate,  etc.) 
Workload  !  “ 

Metric  =  !  T  Performance  Measures 

I  I  (i.e.,  RMS  error,  etc.) 


If  this  is  developed,  then  anyone  performing  research  could 
report  their  results  in  a  standard  format  even  though  not 
all  the  cells  in  the  vector  might  not/could  not  be  measured; 
(g)  use  the  NTEC  document,  discussed  by  Cmdr  Moroney  of 
NPGS,  as  a  baseline  for  future  reports  and  articles.  Then 
new  reports  should  be  abstracted  with  the  keywords  used  in 
the  NTEC  report.  Perhaps  yearly  updates  to  the  NTEC  report 
should  be  planned  for  and  implemented. 
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